{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "In this project, I will use Neural Network Matrix Factorization with Keras to predict the ratings for the movies in [MovieLens Small Datasets](https://grouplens.org/datasets/movielens/latest/)\n",
    "\n",
    "\n",
    "## [Recommender system](https://en.wikipedia.org/wiki/Recommender_system)\n",
    "A recommendation system is basically an information filtering system that seeks to predict the \"rating\" or \"preference\" a user would give to an item. It is widely used in different internet / online business such as Amazon, Netflix, Spotify, or social media like Facebook and Youtube. By using recommender systems, those companies are able to provide better or more suited products/services/contents that are personalized to a user based on his/her historical consumer behaviors\n",
    "\n",
    "\n",
    "## [Neural Collaborative Filtering](https://dl.acm.org/citation.cfm?id=3052569)\n",
    "This notebook will present a general framework named NCF, short for Neural network-based Collaborative Filtering. NCF is generic and can express and generalize matrix factorization under its framework. To supercharge NCF modelling with non-linearities, I will leverage a multi-layer perceptron to learn the user-item interaction function. Extensive experiments on real-world datasets show significant improvements of our proposed NCF framework over the state-of-the-art methods. Empirical evidence shows that using deeper layers of neural networks offers better recommendation performance.\n",
    "\n",
    "\n",
    "## Data Sets\n",
    "We use [MovieLens Small Datasets](https://grouplens.org/datasets/movielens/latest/)\n",
    "This dataset (ml-latest-small) describes 5-star rating and free-text tagging activity from [MovieLens](http://movielens.org), a movie recommendation service. It contains 100004 ratings and 1296 tag applications across 9125 movies. These data were created by 671 users between January 09, 1995 and October 16, 2016. This dataset was generated on October 17, 2016.\n",
    "\n",
    "Users were selected at random for inclusion. All selected users had rated at least 20 movies. No demographic information is included. Each user is represented by an id, and no other information is provided.\n",
    "\n",
    "The data are contained in the files `links.csv`, `movies.csv`, `ratings.csv` and `tags.csv`\n",
    "\n",
    "\n",
    "## Project Content\n",
    "1. Load Data\n",
    "2. Split Data Into Train/Test\n",
    "3. Train Generalized Matrix Factorization Model and Test Model\n",
    "4. Train Multi-Layer Perceptron Model and Test Model\n",
    "5. Train Neural Matrix Factorization Model (NeuMF) and Test Model\n",
    "6. Experiment Observations\n",
    "7. Conclusion and Thoughts\n",
    "\n",
    "NOTE: Train Neural Matrix Factorization Model (NeuMF) is a stacking version of Generalized Matrix Factorization Model and Multi-Layer Perceptron Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# data science imports\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# sklearn imports\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# keras/tensorflow imports\n",
    "from tensorflow.keras.layers import Input, Embedding, Flatten, Dense, Multiply, Concatenate\n",
    "from tensorflow.keras.regularizers import l2\n",
    "from tensorflow.keras.models import Model\n",
    "# from tensorflow.keras.optimizers import Adagrad, Adam, SGD, RMSprop\n",
    "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\n",
    "import tensorflow.keras.backend as K\n",
    "from tensorflow.keras.models import load_model\n",
    "\n",
    "# visualization imports\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# path config\n",
    "data_path = os.path.join(os.environ['DATA_PATH'], 'movie')\n",
    "ratings_filename = 'ratings.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ratings = pd.read_csv(\n",
    "    os.path.join(data_path, ratings_filename),\n",
    "    usecols=['userId', 'movieId', 'rating'],\n",
    "    dtype={'userId': 'int32', 'movieId': 'int32', 'rating': 'float32'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 100004 entries, 0 to 100003\n",
      "Data columns (total 3 columns):\n",
      "userId     100004 non-null int32\n",
      "movieId    100004 non-null int32\n",
      "rating     100004 non-null float32\n",
      "dtypes: float32(1), int32(2)\n",
      "memory usage: 1.1 MB\n"
     ]
    }
   ],
   "source": [
    "df_ratings.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1029</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1061</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1129</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1172</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating\n",
       "0       1       31     2.5\n",
       "1       1     1029     3.0\n",
       "2       1     1061     3.0\n",
       "3       1     1129     2.0\n",
       "4       1     1172     4.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ratings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 671 unique users and 9066 unique movies in this data set\n"
     ]
    }
   ],
   "source": [
    "num_users = len(df_ratings.userId.unique())\n",
    "num_items = len(df_ratings.movieId.unique())\n",
    "print('There are {} unique users and {} unique movies in this data set'.format(num_users, num_items))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 671 distinct users and the max of user ID is also 671\n",
      "There are 9066 distinct movies, however, the max of movie ID is 163949\n",
      "In the context of matrix factorization, the current item vector is in unnecessarily high dimensional space\n",
      "So we need to do some data cleaning to reduce the dimension of item vector back to 9066\n"
     ]
    }
   ],
   "source": [
    "user_maxId = df_ratings.userId.max()\n",
    "item_maxId = df_ratings.movieId.max()\n",
    "print('There are {} distinct users and the max of user ID is also {}'.format(num_users, user_maxId))\n",
    "print('There are {} distinct movies, however, the max of movie ID is {}'.format(num_items, item_maxId))\n",
    "print('In the context of matrix factorization, the current item vector is in unnecessarily high dimensional space')\n",
    "print('So we need to do some data cleaning to reduce the dimension of item vector back to {}'.format(num_items))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reduce_item_dim(df_ratings):\n",
    "    \"\"\"\n",
    "    Reduce item vector dimension to the number of distinct items in our data sets\n",
    "    \n",
    "    input: pd.DataFrame, df_ratings should have columns ['userId', 'movieId', 'rating']\n",
    "    output: pd.DataFrame, df_ratings with new 'MovieID' that is compressed\n",
    "    \"\"\"\n",
    "    # pivot\n",
    "    df_user_item = df_ratings.pivot(index='userId', columns='movieId', values='rating')\n",
    "    # reset movieId\n",
    "    df_user_item = df_user_item.T.reset_index(drop=True).T\n",
    "    # undo pivot/melt - compress data frame\n",
    "    df_ratings_new = df_user_item \\\n",
    "        .reset_index('userId') \\\n",
    "        .melt(\n",
    "            id_vars='userId', \n",
    "            value_vars=df_user_item.columns,\n",
    "            var_name='movieId',\n",
    "            value_name='rating')\n",
    "    # drop nan and final clean up\n",
    "    return df_ratings_new.dropna().sort_values(['userId', 'movieId']).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reduce item dimension before:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1029</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1061</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1129</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1172</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating\n",
       "0       1       31     2.5\n",
       "1       1     1029     3.0\n",
       "2       1     1061     3.0\n",
       "3       1     1129     2.0\n",
       "4       1     1172     4.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('reduce item dimension before:')\n",
    "df_ratings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reduce item dimension after:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>833</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>859</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>906</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>931</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId movieId  rating\n",
       "0       1      30     2.5\n",
       "1       1     833     3.0\n",
       "2       1     859     3.0\n",
       "3       1     906     2.0\n",
       "4       1     931     4.0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ratings = reduce_item_dim(df_ratings)\n",
    "print('reduce item dimension after:')\n",
    "df_ratings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Split Data Into Train/Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape of training data set:\n",
      "(80003, 3)\n",
      "shape of test data set:\n",
      "(20001, 3)\n"
     ]
    }
   ],
   "source": [
    "df_train, df_test = train_test_split(df_ratings, test_size=0.2, shuffle=True, random_state=99)\n",
    "print('shape of training data set:')\n",
    "print(df_train.shape)\n",
    "print('shape of test data set:')\n",
    "print(df_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Train Generalized Matrix Factorization and Test Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### define GMF model architeture and train routine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_GMF_model(num_users, num_items, latent_dim, vu_reg, vi_reg):\n",
    "    \"\"\"\n",
    "    Build Generalized Matrix Factorization Model Topology\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    num_users: int, total number of users\n",
    "    num_iterms: int, total number of items\n",
    "    latent_dim: int, embedded dimension for user vector and item vector\n",
    "    vu_reg: float, L2 regularization of user embedded layer\n",
    "    vi_reg: float, L2 regularization of item embedded layer\n",
    "\n",
    "    Return\n",
    "    ------\n",
    "    A Keras Model with GMF model architeture\n",
    "    \"\"\"\n",
    "    # Input variables\n",
    "    user_input = Input(shape=(1,), dtype='int32', name='user_input')\n",
    "    item_input = Input(shape=(1,), dtype='int32', name='item_input')\n",
    "\n",
    "    MF_Embedding_User = Embedding(\n",
    "        input_dim=num_users + 1,\n",
    "        output_dim=latent_dim,\n",
    "        embeddings_initializer='uniform',\n",
    "        name='user_embedding',\n",
    "        embeddings_regularizer=l2(vu_reg),\n",
    "        input_length=1)\n",
    "    MF_Embedding_Item = Embedding(\n",
    "        input_dim=num_items + 1,\n",
    "        output_dim=latent_dim,\n",
    "        embeddings_initializer='uniform',\n",
    "        name='item_embedding',\n",
    "        embeddings_regularizer=l2(vi_reg),\n",
    "        input_length=1) \n",
    "    \n",
    "    # Crucial to flatten an embedding vector!\n",
    "    user_latent = Flatten()(MF_Embedding_User(user_input))\n",
    "    item_latent = Flatten()(MF_Embedding_Item(item_input))\n",
    "\n",
    "    # Element-wise product of user and item embeddings \n",
    "    predict_vector = Multiply()([user_latent, item_latent])\n",
    "    \n",
    "    # Final prediction layer\n",
    "    prediction = Dense(1, kernel_initializer='glorot_uniform', name='prediction')(predict_vector)\n",
    "    \n",
    "    # Stitch input and output\n",
    "    model = Model([user_input, item_input], prediction)\n",
    "    \n",
    "    return model\n",
    "\n",
    "\n",
    "def train_model(model, learner, batch_size, epochs, val_split, inputs, outputs):\n",
    "    \"\"\"\n",
    "    define training routine, train models and save best model\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    model: a Keras model\n",
    "    learner: str, one of ['sgd', 'adam', 'rmsprop', 'adagrad']\n",
    "    batch_size: num samples per update\n",
    "    epochs: num iterations\n",
    "    val_split: split ratio for validation data\n",
    "    inputs: inputs data\n",
    "    outputs: outputs data\n",
    "    \"\"\"\n",
    "    # add customized metric\n",
    "    def rmse(y_true, y_pred):\n",
    "        return K.sqrt(K.mean(K.square(y_true - y_pred)))\n",
    "    \n",
    "    # compile model\n",
    "    model.compile(optimizer=learner.lower(), loss='mean_squared_error', metrics=['mean_squared_error', rmse])\n",
    "    \n",
    "    # add call backs\n",
    "    early_stopper = EarlyStopping(monitor='val_rmse', patience=10, verbose=1)\n",
    "    model_saver = ModelCheckpoint(filepath=os.path.join(data_path, 'tmp/model.hdf5'),\n",
    "                                  monitor='val_rmse',\n",
    "                                  save_best_only=True,\n",
    "                                  save_weights_only=True)\n",
    "    # train model\n",
    "    history = model.fit(inputs, outputs,\n",
    "                        batch_size=batch_size,\n",
    "                        epochs=epochs,\n",
    "                        validation_split=val_split,\n",
    "                        callbacks=[early_stopper, model_saver])\n",
    "    return history\n",
    "\n",
    "\n",
    "def load_trained_model(model, weights_path):\n",
    "    model.load_weights(weights_path)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### create GMF model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "user_input (InputLayer)         (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "item_input (InputLayer)         (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "user_embedding (Embedding)      (None, 1, 10)        6720        user_input[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "item_embedding (Embedding)      (None, 1, 10)        90670       item_input[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "flatten (Flatten)               (None, 10)           0           user_embedding[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "flatten_1 (Flatten)             (None, 10)           0           item_embedding[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "multiply (Multiply)             (None, 10)           0           flatten[0][0]                    \n",
      "                                                                 flatten_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "prediction (Dense)              (None, 1)            11          multiply[0][0]                   \n",
      "==================================================================================================\n",
      "Total params: 97,401\n",
      "Trainable params: 97,401\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "GMF_model = get_GMF_model(num_users, num_items, 10, 0, 0)\n",
    "GMF_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### train GMF model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Kevin/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:108: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60002 samples, validate on 20001 samples\n",
      "Epoch 1/30\n",
      "60002/60002 [==============================] - 2s 39us/step - loss: 7.2856 - mean_squared_error: 7.2856 - rmse: 2.5509 - val_loss: 1.8228 - val_mean_squared_error: 1.8228 - val_rmse: 1.3466\n",
      "Epoch 2/30\n",
      "60002/60002 [==============================] - 2s 31us/step - loss: 1.2941 - mean_squared_error: 1.2941 - rmse: 1.1303 - val_loss: 1.0716 - val_mean_squared_error: 1.0716 - val_rmse: 1.0314\n",
      "Epoch 3/30\n",
      "60002/60002 [==============================] - 2s 32us/step - loss: 0.9364 - mean_squared_error: 0.9364 - rmse: 0.9632 - val_loss: 0.9376 - val_mean_squared_error: 0.9376 - val_rmse: 0.9641\n",
      "Epoch 4/30\n",
      "60002/60002 [==============================] - 2s 32us/step - loss: 0.8231 - mean_squared_error: 0.8231 - rmse: 0.9029 - val_loss: 0.8891 - val_mean_squared_error: 0.8891 - val_rmse: 0.9384\n",
      "Epoch 5/30\n",
      "60002/60002 [==============================] - 2s 36us/step - loss: 0.7656 - mean_squared_error: 0.7656 - rmse: 0.8704 - val_loss: 0.8667 - val_mean_squared_error: 0.8667 - val_rmse: 0.9263\n",
      "Epoch 6/30\n",
      "60002/60002 [==============================] - 2s 38us/step - loss: 0.7293 - mean_squared_error: 0.7293 - rmse: 0.8494 - val_loss: 0.8583 - val_mean_squared_error: 0.8583 - val_rmse: 0.9217\n",
      "Epoch 7/30\n",
      "60002/60002 [==============================] - 2s 39us/step - loss: 0.7028 - mean_squared_error: 0.7028 - rmse: 0.8338 - val_loss: 0.8539 - val_mean_squared_error: 0.8539 - val_rmse: 0.9191\n",
      "Epoch 8/30\n",
      "60002/60002 [==============================] - 2s 41us/step - loss: 0.6807 - mean_squared_error: 0.6807 - rmse: 0.8202 - val_loss: 0.8528 - val_mean_squared_error: 0.8528 - val_rmse: 0.9186\n",
      "Epoch 9/30\n",
      "60002/60002 [==============================] - 3s 44us/step - loss: 0.6604 - mean_squared_error: 0.6604 - rmse: 0.8082 - val_loss: 0.8550 - val_mean_squared_error: 0.8550 - val_rmse: 0.9197\n",
      "Epoch 10/30\n",
      "60002/60002 [==============================] - 2s 40us/step - loss: 0.6401 - mean_squared_error: 0.6401 - rmse: 0.7957 - val_loss: 0.8605 - val_mean_squared_error: 0.8605 - val_rmse: 0.9227\n",
      "Epoch 11/30\n",
      "60002/60002 [==============================] - 2s 38us/step - loss: 0.6193 - mean_squared_error: 0.6193 - rmse: 0.7818 - val_loss: 0.8673 - val_mean_squared_error: 0.8673 - val_rmse: 0.9264\n",
      "Epoch 12/30\n",
      "60002/60002 [==============================] - 2s 39us/step - loss: 0.5978 - mean_squared_error: 0.5978 - rmse: 0.7682 - val_loss: 0.8765 - val_mean_squared_error: 0.8765 - val_rmse: 0.9312\n",
      "Epoch 13/30\n",
      "60002/60002 [==============================] - 2s 38us/step - loss: 0.5766 - mean_squared_error: 0.5766 - rmse: 0.7547 - val_loss: 0.8886 - val_mean_squared_error: 0.8886 - val_rmse: 0.9376\n",
      "Epoch 14/30\n",
      "60002/60002 [==============================] - 2s 37us/step - loss: 0.5565 - mean_squared_error: 0.5565 - rmse: 0.7413 - val_loss: 0.9025 - val_mean_squared_error: 0.9025 - val_rmse: 0.9449\n",
      "Epoch 15/30\n",
      "60002/60002 [==============================] - 2s 40us/step - loss: 0.5377 - mean_squared_error: 0.5377 - rmse: 0.7285 - val_loss: 0.9127 - val_mean_squared_error: 0.9127 - val_rmse: 0.9502\n",
      "Epoch 16/30\n",
      "60002/60002 [==============================] - 2s 35us/step - loss: 0.5200 - mean_squared_error: 0.5200 - rmse: 0.7162 - val_loss: 0.9266 - val_mean_squared_error: 0.9266 - val_rmse: 0.9575\n",
      "Epoch 17/30\n",
      "60002/60002 [==============================] - 2s 36us/step - loss: 0.5035 - mean_squared_error: 0.5035 - rmse: 0.7051 - val_loss: 0.9397 - val_mean_squared_error: 0.9397 - val_rmse: 0.9642\n",
      "Epoch 18/30\n",
      "60002/60002 [==============================] - 2s 34us/step - loss: 0.4885 - mean_squared_error: 0.4885 - rmse: 0.6942 - val_loss: 0.9541 - val_mean_squared_error: 0.9541 - val_rmse: 0.9716\n",
      "Epoch 00018: early stopping\n"
     ]
    }
   ],
   "source": [
    "# model config\n",
    "BATCH_SIZE = 64\n",
    "EPOCHS = 30\n",
    "VAL_SPLIT = 0.25\n",
    "\n",
    "# train model\n",
    "history = train_model(GMF_model, 'adam', BATCH_SIZE, EPOCHS, VAL_SPLIT, \n",
    "                      inputs=[df_train.userId.values, df_train.movieId.values],\n",
    "                      outputs=df_train.rating.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GMF learning curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_learning_curve(history, metric):\n",
    "    \"\"\"\n",
    "    Plot learning curve to compare training error vs. validation error\n",
    "    \"\"\"\n",
    "    # get training error\n",
    "    errors = history.history[metric]\n",
    "    # get validation error\n",
    "    val_errors = history.history['val_{}'.format(metric)]\n",
    "    # get epochs\n",
    "    epochs = range(1, len(errors) + 1)\n",
    "\n",
    "    # plot\n",
    "    plt.figure(figsize=(12, 7))\n",
    "    plt.plot(epochs, errors, 'bo', label='training {}'.format(metric))\n",
    "    plt.plot(epochs, val_errors, 'b', label='validation {}'.format(metric))\n",
    "    plt.xlabel('number of epochs')\n",
    "    plt.ylabel(metric)\n",
    "    plt.title('Model Learning Curve')\n",
    "    plt.grid(True)\n",
    "    plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtoAAAG5CAYAAACwZpNaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3XmYXGWd9//3NwuELELYYliyiKgsCQlkWB4QOoxiwAUGwQfMKKIYQcBlcGcEdQyPPuKGCgwql48zgQyD4PCTOApKA44JEmJACLInEMMaSMhCgCbf3x+nOlSa7k6n06equvv9uq5zddU59zn1rTtF88md+9wVmYkkSZKknjWg3gVIkiRJfZFBW5IkSSqBQVuSJEkqgUFbkiRJKoFBW5IkSSqBQVuSJEkqgUFbknpQRIyLiIyIQV1o+6GI+EMt6toSEbE6It5Q7zokqbcxaEvqtyJicUS8FBE7ttm/sBKWx9Wnss0L7GXLzOGZ+XAZ146IN0XEf0bEMxGxMiLuioh/ioiBZbyeJNWSQVtSf/cIcHLrk4iYAGxTv3Jqq56BNiL2AG4DHgMmZOa2wInAFGBEN65X97+USFI1g7ak/u7fgA9WPT8F+Hl1g4jYNiJ+HhFPR8SSiPjniBhQOTYwIi6sjMg+DLyznXN/GhGPR8TfIuLrWxpuI2JARHwhIh6KiOURcVVEbF91/D8j4onKCPEtEbFP1bGfRcQlETEnItYAUyv7fhQR10fEqoi4rRKCW8/JiHhj1fmdtT0qIu6rvPbFEXFzRJzWwVv5KvDHzPynzHwcIDPvy8z3Z+aKiGiKiKVt3vviiHhb5fFXIuLqiPj3iHge+FJEvNCmLyZX/mwGV55/OCLujYjnIuI3ETG2+38SktQ5g7ak/m4e8LqI2KsSgP838O9t2vwA2BZ4A3AERTA/tXLso8C7gMkUI7EntDn3/wEtwBsrbY4COgqeXfUJ4LhKLbsAzwE/qjr+a2BPYGdgATCrzfnvB2ZSjBq3zhE/mSL4jgQerBzvSLttK1Nwrga+COwA3Af8r06u87ZK+y1xbOUa2wHfAuYC7606/n7g6sx8OSKOA74EHA/sBNwKXLmFry9JHTJoS9Kro9pvB/4K/K31QFX4/mJmrsrMxcC3gQ9UmrwP+F5mPpaZzwL/p+rcUcDRwKcyc01mPgV8FzhpC+v9GHBuZi7NzBeBrwAntE6dyMzLK7W2HtsvIratOv+/MvN/MnN9Zq6r7LsmM/+UmS0UwXxSJ6/fUdtjgHsy85rKsYuAJzq5zg7A45vzxtsxNzN/WXkvLwBXUJkKFBFB0ddXVNp+DPg/mXlvpb4LgEmOaksqi/PZJKkI2rcA42kzbQTYEdgKWFK1bwmwa+XxLhRzjKuPtRoLDAYeLzIfUAxwVLfvjrHAtRGxvmrfK8CoiHiCYoT5RIpR29Y2OwIrK4/be/3qQLwWGN7J63fUdqO+yMxsO/WjjeXA6E6Od0Xb93I18IOI2IViVD8pRq6h6LfvR8S3q9oHxZ/lEiSphzmiLanfy8wlFDdFHgNc0+bwM8DLFCGt1RheHfV+HNi9zbFWjwEvAjtm5naV7XWZuQ9b5jHg6KprbpeZQzLzbxRTJY6lmJaxLTCuck5UnZ9b+PodeRzYrfVJZUR5t46bcyMbT/Noaw0wtOp6Ayn+8lBto/eSmSuA31L8S8P7gSszs7XNY8DH2vTbNpn5x87fliR1j0FbkgofAY7MzDXVOzPzFeAqYGZEjKhMM/gnXp3HfRXwiYjYLSJGAl+oOvdxitD37Yh4XeUmxj0i4ojNqGvriBhStQ0ALq3UMxYgInaKiGMr7UdQhPvlFCH1gs3rhi1yPTAhIo6rTGM5E3h9J+3PB/5XRHwrIl4PEBFvrNzcuB1wPzAkIt5ZuZnxn4Gtu1DHFRRTgd7Lq9NGoOi3L7beHFq5UfXEzXyPktRlBm1JAjLzocyc38HhsylGVx+muHnwCuDyyrEfA78B7qS48bDtiPgHKaaeLKK4afFqNm+6xGrghartSOD7wHXAbyNiFcUNnQdV2v+cYhrE3yqvOW8zXmuLZOYzFFNW/i9F0N8bmE8R/Ntr/xBwCMWo+z0RsRL4ReWcVZm5Evg48BOK97MG6GwqSqvrKKaNPJmZd1a93rXAN4HZlVVK7qaYQy9JpYhX/0VNkqSeUxl9XwpMz8yb6l2PJNWaI9qSpB4TEe+IiO0iYmuKpfSCGo6qS1IjMWhLknrSIcBDFDeRvhs4rrLsniT1O04dkSRJkkrgiLYkSZJUgj71hTU77rhjjhs3rt5l9Blr1qxh2LBh9S6jT7OPa8N+Lp99XBv2c/ns49ro7f18xx13PJOZbdf1f40+FbTHjRvH/Pkdrc6lzdXc3ExTU1O9y+jT7OPasJ/LZx/Xhv1cPvu4Nnp7P0dEl75N1qkjkiRJUgkM2pIkSVIJDNqSJElSCfrUHG1JkqRG8vLLL7N06VLWrVtX71Iayrbbbsu9995b7zI2aciQIey2224MHjy4W+cbtCVJkkqydOlSRowYwbhx44iIepfTMFatWsWIESPqXUanMpPly5ezdOlSxo8f361rOHVEkiSpJOvWrWOHHXYwZPdCEcEOO+ywRf8aYdCWJEkqkSG799rSPzuDtiRJklQCg7YkSVIftWLFCi6++OJunXvMMcewYsWKTtucd9553Hjjjd26fn9g0JYkSWoQs2bBuHEwYEDxc9asLbteZ0H7lVde6fTcOXPmsN1223Xa5mtf+xpve9vbul1fRzZVW29h0JYkSWoAs2bBjBmwZAlkFj9nzNiysP2FL3yBhx56iEmTJvHZz36W5uZmpk6dyvvf/34mTJgAwHHHHccBBxzAPvvsw2WXXbbh3HHjxvHMM8+wePFi9tprLz760Y+yzz77cNRRR/HCCy8A8KEPfYirr756Q/vzzz+f/fffnwkTJvDXv/4VgKeffpq3v/3t7L///nzsYx9j7NixLF++/DW1Dh8+nPPOO4+DDjqIuXPnMm7cOL70pS9xyCGHMGXKFBYsWMA73vEO9thjDy699FIAHn/8cQ4//HAmTZrEvvvuy6233grAb3/7Ww455BD2339/TjzxRFavXt39TtwCBm1JkqQGcO65sHbtxvvWri32d9c3vvEN9thjDxYuXMi3vvUtAP70pz8xc+ZMFi1aBMDll1/OHXfcwfz587nooovaDcEPPPAAZ555Jvfccw/bbbcdv/jFL9p9vR133JEFCxZwxhlncOGFFwLw1a9+lSOPPJIFCxbwD//wDzz66KPtnrtmzRr23XdfbrvtNg477DAAdt99d+bOnctb3/rWDaF+3rx5nHfeeQBcccUVvOMd72DhwoXceeedTJo0iWeeeYavf/3r3HjjjSxYsIApU6bwne98p/uduAVcR1uSJKkBdJA/O9zfXQceeOBG60JfdNFFXHvttQA89thjPPDAA+ywww4bnTN+/HgmTZoEwAEHHMDixYvbvfbxxx+/oc0111wDwB/+8IcN1582bRojR45s99yBAwfy3ve+d6N973nPewCYMGECq1evZsSIEYwYMYIhQ4awYsUK/u7v/o4Pf/jDvPzyyxx33HFMmjSJm2++mUWLFnHooYcC8NJLL3HIIYd0uX96kiPaW6Cn51FJkqT+a8yYzdvfXcOGDdvwuLm5mRtvvJG5c+dy5513Mnny5HbXjd566603PB44cCAtLS3tXru1XXWbzOxSXUOGDGHgwIHtXm/AgAEb1TBgwABaWlo4/PDDueWWW9h11135wAc+wM9//nMyk7e//e0sXLiQhQsXsmjRIn760592qYaeZtDupjLmUUmSpP5r5kwYOnTjfUOHFvu7a8SIEaxatarD4ytXrmTkyJEMHTqUv/71r8ybN6/7L9aBww47jKuuugoo5k4/99xzPXbtJUuWsPPOO/PRj36Uj3zkIyxYsICDDz6Y//mf/+HBBx8EYO3atdx///099pqbw6DdTWXMo5IkSf3X9Olw2WUwdixEFD8vu6zY31077LADhx56KPvuuy+f/exnX3N82rRptLS0MHHiRL785S9z8MEHb8E7aN/555/Pb3/7W/bff39+/etfM3r0aIYPH94j125ubmbSpElMnjyZX/ziF3zyk59kp5124mc/+xknn3wyEydO5OCDD95wY2atRVeH83uDKVOm5Pz582vyWgMGFCPZbUXA+vU1KaF0zc3NNDU11buMPs0+rg37uXz2cW3Yz+Xr6T6+99572WuvvXrser3Riy++yMCBAxk0aBBz587ljDPO4NZbb2XEiBH1Lq1L2vszjIg7MnPKps71ZshuGjOmmC7S3n5JkiQVHn30Ud73vvexfv16ttpqK3784x/Xu6SaMWh308yZxZzs6ukjWzqPSpIkqa/Zc889+fOf/7zRvs7mjfclztHupjLmUUmSJKnvcER7C0yfbrCWJElS+xzRliRJkkpQWtCOiN0j4qaIuDci7omIT7bTpikiVkbEwsp2XtWxaRFxX0Q8GBFfKKtOSZIkqQxljmi3AOdk5l7AwcCZEbF3O+1uzcxJle1rABExEPgRcDSwN3ByB+dKkiSpB7Wucb1s2TJOOOGEdts0NTWxqSWVv/e977G2atWIY445hhUrVvRcob1AaUE7Mx/PzAWVx6uAe4Fdu3j6gcCDmflwZr4EzAaOLadSSZIktbXLLrtw9dVXd/v8tkF7zpw5bLfddj1R2gYdfRV8o6jJzZARMQ6YDNzWzuFDIuJOYBnwmcy8hyKQP1bVZilwUAfXngHMABg1ahTNzc09Vnd/t3r1avuzZPZxbdjP5bOPa8N+Ll9P9/G2225b16XszjvvPHbffXc++tGPAnDBBRcwYsQITj31VE4++WRWrFjByy+/zJe//GXe+c53bjhv1apVLFmyhPe9733cdtttvPDCC5xxxhncd999vPnNb2b16tWsWbOGVatW8elPf5oFCxbwwgsvcOyxx3LuuedyySWXsGzZMo444gh22GEHrr/+evbdd19uvvlmdthhBy666CJmzZoFwAc/+EHOPPNMlixZwnvf+14OOeQQbrvtNkaPHs3s2bPZZpttNnpPp59+OiNHjuSuu+5iv/32Y/jw4SxZsoQnnniChx56iAsuuIDbb7+dG264gdGjR3PVVVcxePBgzj//fObMmcOgQYM48sgjmTlzJs888wyf+tSneOyxInZ+85vffM23Y65bt677n4nMLHUDhgN3AMe3c+x1wPDK42OAByqPTwR+UtXuA8APNvVaBxxwQKrn3HTTTfUuoc+zj2vDfi6ffVwb9nP5erqPFy1atOHxJz+ZecQRPbt98pOdv/6CBQvy8MMP3/B8r732yiVLluTLL7+cK1euzMzMp59+OvfYY49cv359ZmYOGzYsMzMfeeSR3GeffTIz89vf/naeeuqpmZl555135sCBA/P222/PzMzly5dnZmZLS0seccQReeedd2Zm5tixY/Ppp5/e8Nqtz+fPn5977713rl69OletWpV77713LliwIB955JEcOHBg/vnPf87MzBNPPDH/7d/+7TXv6ZRTTsl3vvOd2dLSkpmZ559/fh566KH50ksv5cKFC3ObbbbJOXPmZGbmcccdl9dee20uX7483/SmN214j88991xmZp588sl56623ZmbmkiVL8i1vectrXq/6z7AVMD+7kINLXXUkIgYDvwBmZeY17YT85zNzdeXxHGBwROxIMYK9e1XT3ShGvCVJktRFkydP5qmnnmLZsmXceeedjBw5kjFjxpCZfOlLX2LixIm87W1v429/+xtPPvlkh9e55ZZb+Md//EcAJk6cyMSJEzccu+qqq9h///2ZPHky99xzD4sWLeq0pj/84Q+8613vYtiwYQwfPpzjjz+eW2+9FYDx48czadIkAA444AAWL17c7jVOPPFEBg4cuOH50UcfzeDBg5kwYQKvvPIK06ZNA2DChAksXryY173udQwZMoTTTjuNa665hqFDhwJw4403ctZZZzFp0iTe85738Pzzz/fov0CUNnUkIgL4KXBvZn6ngzavB57MzIyIAynmjC8HVgB7RsR44G/AScD7y6pVkiSpbN/7Xn1e94QTTuDqq6/miSee4KSTTgJg1qxZPP3009xxxx0MHjyYcePGsW7duk6vU0S7jT3yyCNceOGF3H777YwcOZIPfehDm7xOMSDcvq233nrD44EDB/LCCy+0227YsGHtnjdgwAAGDx68odYBAwbQ0tLCoEGD+NOf/sTvfvc7Zs+ezQ9/+EN+//vfs379eubOnfua6Sk9pcwR7UMppnwcWbV83zERcXpEnF5pcwJwd2WO9kXASZUR+RbgLOA3FDdRXpXF3G1JkiRthpNOOonZs2dz9dVXb1hFZOXKley8884MHjyYm266iSVLlnR6jcMPP3zDnOq7776bu+66C4Dnn3+eYcOGse222/Lkk0/y61//esM5I0aMaHd0+PDDD+f6669n7dq1rFmzhmuvvZa3vvWtPfV227V69WpWrlzJMcccw/e+9z0WLlwIwFFHHcUPf/jDDe1a9/eU0ka0M/MPwGv/6rNxmx8CP+zg2BxgTgmlSZIk9Rv77LMPq1atYtddd2X06NEATJ8+nXe/+91MmTKFSZMm8Za3vKXTa5xxxhmceuqpTJw4kUmTJnHggQcCsN9++zF58mT22Wcf3vCGN3DooYduOGfGjBkcffTRjB49mptuumnD/v3335/p06dvuMZpp53G5MmTO5wm0hNWrVrFsccey7p168hMvvvd7wJw0UUXceaZZzJx4kRaWlo4/PDDufTSS3vsdaOz4fveZsqUKbmpNR3Vdc3NzTQ1NdW7jD7NPq4N+7l89nFt2M/l6+k+vvfee9lrr7167Hp9xapVqxgxYkS9y+iS9v4MI+KOzJyyqXP9CnZJkiSpBAZtSZIkqQQGbUmSpBL1pWm6/c2W/tkZtCVJkkoyZMgQli9fbtjuhTKT5cuXM2TIkG5foyZfwS5JktQf7bbbbixdupSnn3663qU0lHXr1m1RgK2VIUOGsNtuu3X7fIO2JElSSQYPHsz48ePrXUbDaW5uZvLkyfUuo3ROHZEkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKUFrQjojdI+KmiLg3Iu6JiE+202Z6RNxV2f4YEftVHVscEX+JiIURMb+sOiVJkqQyDCrx2i3AOZm5ICJGAHdExA2ZuaiqzSPAEZn5XEQcDVwGHFR1fGpmPlNijZIkSVIpSgvamfk48Hjl8aqIuBfYFVhU1eaPVafMA3Yrqx5JkiSpliIzy3+RiHHALcC+mfl8B20+A7wlM0+rPH8EeA5I4F8z87IOzpsBzAAYNWrUAbNnz+7x+vur1atXM3z48HqX0afZx7VhP5fPPq4N+7l89nFt9PZ+njp16h2ZOWVT7UoP2hExHLgZmJmZ13TQZipwMXBYZi6v7NslM5dFxM7ADcDZmXlLZ681ZcqUnD/f6dw9pbm5maampnqX0afZx7VhP5fPPq4N+7l89nFt9PZ+joguBe1SVx2JiMHAL4BZnYTsicBPgGNbQzZAZi6r/HwKuBY4sMxaJUmSpJ5U5qojAfwUuDczv9NBmzHANcAHMvP+qv3DKjdQEhHDgKOAu8uqVZIkSeppZa46cijwAeAvEbGwsu9LwBiAzLwUOA/YAbi4yOW0VIbhRwHXVvYNAq7IzP8usVZJkiSpR5W56sgfgNhEm9OA09rZ/zCw32vPkCRJknoHvxlSkiRJKoFBW5IkSSqBQVuSJEkqgUFbkiRJKoFBW5IkSSqBQVuSJEkqgUFbkiRJKoFBW5IkSSqBQVuSJEkqgUFbkiRJKoFBW5IkSSqBQVuSJEkqgUFbkiRJKoFBW5IkSSqBQVuSJEkqgUFbkiRJKoFBW5IkSSqBQVuSJEkqgUFbkiRJKoFBW5IkSSqBQVuSJEkqgUFbkiRJKoFBW5IkSSqBQVuSJEkqgUFbkiRJKoFBW5IkSSqBQVuSJEkqgUFbkiRJKoFBW5IkSSqBQVuSJEkqgUFbkiRJKoFBW5IkSSqBQVuSJEkqgUFbkiRJKoFBW5IkSSqBQVuSJEkqgUFbkiRJKkFpQTsido+ImyLi3oi4JyI+2U6biIiLIuLBiLgrIvavOnZKRDxQ2U4pq05JkiSpDINKvHYLcE5mLoiIEcAdEXFDZi6qanM0sGdlOwi4BDgoIrYHzgemAFk597rMfK7EeiVJkqQeU9qIdmY+npkLKo9XAfcCu7Zpdizw8yzMA7aLiNHAO4AbMvPZSri+AZhWVq2SJElSTytzRHuDiBgHTAZua3NoV+CxqudLK/s62t/etWcAMwBGjRpFc3NzT5QsYPXq1fZnyezj2rCfy2cf14b9XD77uDb6Sz+XHrQjYjjwC+BTmfl828PtnJKd7H/tzszLgMsApkyZkk1NTd0vVhtpbm7G/iyXfVwb9nP57OPasJ/LZx/XRn/p51JXHYmIwRQhe1ZmXtNOk6XA7lXPdwOWdbJfkiRJ6hXKXHUkgJ8C92bmdzpodh3wwcrqIwcDKzPzceA3wFERMTIiRgJHVfZJkiRJvUKZU0cOBT4A/CUiFlb2fQkYA5CZlwJzgGOAB4G1wKmVY89GxL8At1fO+1pmPltirZIkSVKPKi1oZ+YfaH+udXWbBM7s4NjlwOUllCZJkiSVzm+GlCRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKYNCWJEmSSmDQliRJkkpg0JYkSZJKMKisC0fE5cC7gKcyc992jn8WmF5Vx17ATpn5bEQsBlYBrwAtmTmlrDolSZKkMpQ5ov0zYFpHBzPzW5k5KTMnAV8Ebs7MZ6uaTK0cN2RLkiSp1yktaGfmLcCzm2xYOBm4sqxaJEmSpFqLzCzv4hHjgF+1N3Wkqs1QYCnwxtYR7Yh4BHgOSOBfM/OyTs6fAcwAGDVq1AGzZ8/usfr7u9WrVzN8+PB6l9Gn2ce1YT+Xzz6uDfu5fPZxbfT2fp46deodXZl1Udoc7c3wbuB/2kwbOTQzl0XEzsANEfHXygj5a1RC+GUAU6ZMyaamptIL7i+am5uxP8tlH9eG/Vw++7g27Ofy2ce10V/6uRFWHTmJNtNGMnNZ5edTwLXAgXWoS5IkSeq2ugbtiNgWOAL4r6p9wyJiROtj4Cjg7vpUKEmSJHVPmcv7XQk0ATtGxFLgfGAwQGZeWmn2D8BvM3NN1amjgGsjorW+KzLzv8uqU5IkSSpDaUE7M0/uQpufUSwDWL3vYWC/cqqSJEmSaqMR5mhLkiRJfY5BW5IkSSqBQVuSJEkqgUFbkiRJKoFBW5IkSSqBQVuSJEkqgUFbkiRJKoFBW5IkSSpBl4J2FP4xIs6rPB8TEQeWW5okSZLUe3V1RPti4BCg9dseVwE/KqUiSZIkqQ/o6lewH5SZ+0fEnwEy87mI2KrEuiRJkqRerasj2i9HxEAgASJiJ2B9aVVJkiRJvVxXg/ZFwLXAzhExE/gDcEFpVUmSJEm9XJemjmTmrIi4A/h7IIDjMvPeUiuTJEmSerGurjqyB/BIZv4IuBt4e0RsV2plkiRJUi/W1akjvwBeiYg3Aj8BxgNXlFaVJEmS1Mt1NWivz8wW4Hjg+5n5aWB0eWVJkiRJvdvmrDpyMvBB4FeVfYPLKUmSJEnq/boatE+l+MKamZn5SESMB/69vLIkSZKk3q2rq44sAj5R9fwR4BtlFSVJkiT1dl1ddeRdEfHniHg2Ip6PiFUR8XzZxUmSJEm9VVe/gv17FDdC/iUzs8R6JEmSpD6hq3O0HwPuNmRLkiRJXdPVEe3PAXMi4mbgxdadmfmdUqqSJEmSermuBu2ZwGpgCLBVeeVIkiRJfUNXg/b2mXlUqZVIkiRJfUhX52jfGBEGbUmSJKmLNhm0IyIo5mj/d0S84PJ+kiRJ0qZtcupIZmZELMzM/WtRkCRJktQXdHXqyNyI+LtSK5EkSZL6kK7eDDkVOD0iFgNrgKAY7J5YVmGSJElSb9bVoH10qVVIkiRJfUyXgnZmLim7EEmSJKkv6eocbUmSJEmbwaAtSZIklcCgLUmSJJXAoC1JkiSVoLSgHRGXR8RTEXF3B8ebImJlRCysbOdVHZsWEfdFxIMR8YWyapQkSZLKUuaI9s+AaZtoc2tmTqpsXwOIiIHAjyiWFNwbODki9i6xTkmSJKnHlRa0M/MW4NlunHog8GBmPpyZLwGzgWN7tDhJkiSpZJGZ5V08Yhzwq8zct51jTcAvgKXAMuAzmXlPRJwATMvM0yrtPgAclJlndfAaM4AZAKNGjTpg9uzZJbyT/mn16tUMHz683mX0afZxbdjP5bOPa8N+Lp99XBu9vZ+nTp16R2ZO2VS7rn4zZBkWAGMzc3VEHAP8EtiT4uvd2+rwbwOZeRlwGcCUKVOyqamphFL7p+bmZuzPctnHtWE/l88+rg37uXz2cW30l36u26ojmfl8Zq6uPJ4DDI6IHSlGuHevarobxYi3JEmS1GvULWhHxOsjIiqPD6zUshy4HdgzIsZHxFbAScB19apTkiRJ6o7Spo5ExJVAE7BjRCwFzgcGA2TmpcAJwBkR0QK4Sre9AAAe4klEQVS8AJyUxYTxlog4C/gNMBC4PDPvKatOSZIkqQylBe3MPHkTx38I/LCDY3OAOWXUJUmSJNWC3wwpSZIklcCgLUmSJJXAoC1JkiSVwKAtSZIklcCgLUmSJJXAoC1JkiSVwKAtSZIklcCgLUmSJJXAoC1JkiSVwKAtSZIklcCgLUmSJJXAoC1JkiSVwKAtSZIklcCgLUmSJJXAoC1JkiSVwKAtSZIklcCgLUmSJJXAoC1JkiSVwKAtSZIklcCgLUmSJJXAoC1JkiSVwKDdAzLrXYEkSZIajUF7C2TCmWfCGWfUuxJJkiQ1GoP2FoiAbbaBf/1XmDev3tVIkiSpkRi0t9BXvgK77lqMare01LsaSZIkNQqD9hYaPhy+/31YuBAuvrje1UiSJKlRGLR7wPHHw7Rp8M//DI8/Xu9qJEmS1AgM2j0gAn7wA3jpJTjnnHpXI0mSpEZg0O4hb3wjfPGLcOWV8Lvf1bsaSZIk1ZtBuwd9/vOwxx7w8Y/Diy/WuxpJkiTVk0G7Bw0ZAj/8Idx/P1x4Yb2rkSRJUj0ZtHvYtGlwwgnw9a/DI4/UuxpJkiTVi0G7BN/9LgwcCGef7dezS5Ik9VcG7RLstht89atw/fVw3XX1rkaSJEn1YNAuySc+ARMmFD/XrKl3NZIkSao1g3ZJBg8uviny0UeL+dqSJEnqXwzaJTrsMDj11GIFkkWL6l2NJEmSasmgXbJvfhNGjIAzz/TGSEmSpP6ktKAdEZdHxFMRcXcHx6dHxF2V7Y8RsV/VscUR8ZeIWBgR88uqsRZ22gm+8Q1oboZZs+pdjSRJkmqlzBHtnwHTOjn+CHBEZk4E/gW4rM3xqZk5KTOnlFRfzZx2Ghx4IJxzDqxYUe9qJEmSVAulBe3MvAV4tpPjf8zM5ypP5wG7lVVLvQ0YAJdcAs88A//8z/WuRpIkSbUQWeLE4YgYB/wqM/fdRLvPAG/JzNMqzx8BngMS+NfMbDvaXX3uDGAGwKhRow6YPXt2zxRfgosueiO//OWuXHLJHbz5zavrXc4mrV69muHDh9e7jD7NPq4N+7l89nFt2M/ls49ro7f389SpU+/oyqyLugftiJgKXAwclpnLK/t2ycxlEbEzcANwdmWEvFNTpkzJ+fMbd0r3ypXwlrfA7rvD3LnFt0c2submZpqamupdRp9mH9eG/Vw++7g27Ofy2ce10dv7OSK6FLTruupIREwEfgIc2xqyATJzWeXnU8C1wIH1qbBnbbstfOc7cPvt8OMf17saSZIklaluQTsixgDXAB/IzPur9g+LiBGtj4GjgHZXLumNTjoJjjwSvvhFeOqpelcjSZKkspS5vN+VwFzgzRGxNCI+EhGnR8TplSbnATsAF7dZxm8U8IeIuBP4E3B9Zv53WXXWWgT86EfF17J/7nP1rkaSJEllGVTWhTPz5E0cPw04rZ39DwP7vfaMvuMtb4HPfhYuuAA+/GE4/PB6VyRJkqSe5jdD1sm558LYsXDGGfDyy/WuRpIkST3NoF0nQ4fCD34AixbB975X72okSZLU0wzadfTud8N73gNf+Qo89li9q5EkSVJPMmjX2fe/D5nwqU/VuxJJkiT1JIN2nY0bB1/+MlxzDcyZU+9qJEmS1FMM2g3gnHOKlUjOPhteeKHe1UiSJKknGLQbwFZbwcUXw8MPwze+Ue9qJEmS1BMM2g1i6lSYPr0I2g88UO9qJEmStKUM2g3kwgthyBA488ziBklJkiT1XgbtBvL618PMmXDDDfCf/1nvaiRJkrQlDNoN5owzYPLkYrm/55+vdzWSJEnqLoN2gxk4EC65BJ54ovgiG0mSJPVOBu0GdNBBMGMGXHQR3HlnvauRJElSdxi0G9QFF8DIkfDxj8P69V07Z9as4gtwBgwofs6aVWaFkiRJ6oxBu0Ftvz1861vwxz/Cz3626fazZhWj4EuWFCuWLFlSPDdsS5Ik1YdBu4Gdcgq89a3wuc/B8uWdtz33XFi7duN9a9cW+yVJklR7Bu0GFlF8Y+SKFfCFL3Te9tFHN2+/JEmSymXQbnD77guf/jT85Ccwd27H7caM2bz9kiRJKpdBuxc4/3zYdddije2WlvbbzJwJQ4duvG/o0GK/JEmSas+g3QsMHw7f/36x1N+PftR+m+nT4bLLYOzYYsrJ2LHF8+nTa1urJEmSCgbtXuL442HaNPjyl2HZsvbbTJ8OixcXywEuXmzIliRJqieDdi8RAT/4Abz0EpxzTr2rkSRJ0qYYtHuRN74RvvhFmD0bbryx3tVIkiSpMwbtXubzny8C95lnwosv1rsaSZIkdcSg3csMGVLcEHn//XDhhfWuRpIkSR0xaPdCRx0FJ54IX/86PPxwvauRJElSewzavdR3vwuDBsHZZ0NmvauRJElSWwbtXmrXXeGrX4U5c+C//qve1UiSJKktg3YvdvbZMGECfOITsGZNvauRJElSNYN2LzZ4MFx8MTz2GPzLv9S7GkmSJFUzaPdyhx0Gp54K3/42LFpU72okSZLUyqDdB3zzmzBiBHz8494YKUmS1CgM2n3ATjvBN74BN98Ms2bVuxpJkiSBQbvPOO00OOggOOcceO65elcjSZIkg3YfMWAAXHIJPPMM7L8/nHsu3HNPvauSJEnqvwzafcjkyfDLX8Kb3lRMJdl3X5g4sXi8eHG9q5MkSepfDNp9zLvfDb/5DSxbBj/4AQwfDl/8IowfD4ceCj/6ETz1VL2rlCRJ6vtKDdoRcXlEPBURd3dwPCLiooh4MCLuioj9q46dEhEPVLZTyqyzLxo1Cs46C/74R3j4YbjgAnj++WLfLrvAtGnw858X+yRJktTzyh7R/hkwrZPjRwN7VrYZwCUAEbE9cD5wEHAgcH5EjCy10j5s/PhiVPsvf4G77oLPfQ7uuw9OOaUI5CeeCNdcA+vW1btSSZKkvqPUoJ2ZtwDPdtLkWODnWZgHbBcRo4F3ADdk5rOZ+RxwA50HdnXRhAnF6PbDDxej3aedBrfcAu99bxG6P/xhuOEGaGmpd6WSJEm9W73naO8KPFb1fGllX0f71UMi4JBDinncf/tbMa/7+OPh6qvhqKNgt93gooveyLx5fgmOJElSd0SWnKIiYhzwq8zct51j1wP/JzP/UHn+O+BzwJHA1pn59cr+LwNrM/Pb7VxjBsW0E0aNGnXA7NmzS3on/cOLLw5g3rzt+f3vRzF37va8/PJARo9+gSOPfIq///snGT9+bb1L7FNWr17N8OHD611Gn2c/l88+rg37uXz2cW309n6eOnXqHZk5ZVPt6h20/xVozswrK8/vA5pat8z8WHvtOjJlypScP39+T5bfr/3qV7eyfPlbueIKuPFGWL++mHry/vfDSSfBuHH1rrD3a25upqmpqd5l9Hn2c/ns49qwn8tnH9dGb+/niOhS0K731JHrgA9WVh85GFiZmY8DvwGOioiRlZsgj6rsUw0NH/4Kp5zicoGSJEndUfbyflcCc4E3R8TSiPhIRJweEadXmswBHgYeBH4MfBwgM58F/gW4vbJ9rbJPddLecoGrVnW+XOCsWcWo94ABxc9Zs+pVvSRJUu0NKvPimXnyJo4ncGYHxy4HLi+jLm2Z1uUCW5cMvPLKYjvlFBgyBN71Lth1V7jsMnjhheKcJUtgxozi8fTp9atdkiTVRia8+GKRBdau3Xi7447tWLWq/WNr13a8v+2xz38ezjmn3u+0Y6UGbfV9EyYU28yZMG9eEbj/4z/an06ydi186UsGbUmS6mX9+uJ7M154oftbVwJw69bxrYCTOqxxq61g6NBi22abVx8PHQqjR298bO+9S+mmHmPQVo9oXS7wkEPgO9+BwYPbb/foo7D77rDHHvCGN7y6tT7fccfiWpIk9QetwbdtUG37uCe2deuKEebu2pwAXH2svf333fdnDj108muObbMNDOpD6bQPvRU1ikGDYOzYYrpIW9tuC0ceWczzbr3Jstrw4RsH7+ogPnZs8R+5JEllyoSXX9448HYWgts+3py23f1W5ogilHa07bxz58c3dxsyBAYO7Lk+HjJkJVM2uWZH72fQVilmzizmZK+tWnZ76NBilZLqqSNr18LixUXwfvhheOih4ud998Gvf73xL6ABA4ov0ukoiG+/vaPhktRbrV8PL71UjLi2jrx2ZevptmvWHMZLL8Err2z+e4hof/R2m21gxIgi/HZ0vL2R3daf7W1bbeX/83oDg7ZK0Rqmzz23mC4yZkwRvtvOzx46tJhf1d4cq/Xr4YknXg3h1UH8+uuLY9Ve97rXTkVp3caO7Xg6iySpfdXTGro6Uru5I7pr1xYB9+WXe67urbaCrbfueBsypPgX1vaOPf3047zpTbtvdiAeOtTwq9cyaKs006dv2Y2PAwYUSwfusgscdthrj69ZA4888togvmhREcSr56ENGFCE/XHjipHvkSNhu+02vQ0b5i9NSY0hE1paguef3/hmttbHm/OzqwG4u9MaOpvLO2rUxsdapyV0Foo7C81t225p2G1ufoimpt27fwGpikFbvdawYbDvvsXW1vr18Pjjr46At25LlhTTUlasKLY1azp/jUGDNh3GOwvt22xjUJd6u8xiGkHrtIb2fm5qX3fCcHs/168/otvvY+DAV0Nt2xA8fDjstNOmpzV0dYS3J+fySr2ZQVt90oABxVreu+4Khx/ecbuXX4aVK18N3s899+rjjrZly15t17pOeEcGD+48jD/zzFhuv70YgWndtt564+ebc2zwYIO9GtsrrxT/3bW0FD+3ZNvUNTYnFG/qWMdLlG2+QYOKsNsaetv+3GGH1+5vffzEEw+z115v6PDczn72pZUcpN7C/+zUrw0eXCwpuOOO3Tv/xRc3Duptt/aC+6OPvvp43brxPfuGKN5TVwN62/2DBxcjUYMGFT+rH5e1r7NjAyrfXRux8dZ236baPPvsVjzxxKavs6lrZxb/WtL255bs25JrtLQU2yuvvPp4U1tZbVesmMLWW286HPdkYO1MxKvTCdp+1qv3bb11cZNa23Ydte/Osa222jj0bkngbW5+lKamN/RcR0kqlUFb2gJbb13cRb7zzt07/8Ybb+aQQ47YMJJWvVWPsJW1f9Wq1+5/5ZVXA1bbxy0tRcDrff5XvQtoSIMHF6Gv7db6l51NbUOHvtp2m23WMXr0cAYPptNt0KDOj3d129R1nLogqREYtNUvzJq16RVQ6mHQoGTYsGK+eW9RPZraWSDf1LGu7Mt87dZaQ0fP29t3//33s+eeb9rs89o+HzCgGCmt/tnVfZvbviv7uhKGOwrPrf9a0FOam++mqampZy8qSb2cQVt93qxZG6/pvWRJ8RwaI2z3NhGvTvXoLZqbl9HU9KZ6lyFJ6md6eExDajznnrvxF+dA8fzcc+tTjyRJ6h8M2urzHn108/ZLkiT1BIO2+rwxYzZvvyRJUk8waKvPmzmzWB2h2tChxX5JkqSyGLTV502fDpddBmPHFjfyjR1bPK/njZCzZhVfB3/kkUcwblzxXJIk9S2uOqJ+Yfr0xllhZONVUMJVUCRJ6qMc0ZZqzFVQJEnqHwzaUo25CookSf2DQVuqsUZcBaV1zviAAThnXJKkHmLQlmqs0VZBaZ0zvmRJ8TXjrXPGDduSJG0Zg7ZUYxuvgpJ1XwXFOeOSJJXDoC3VwfTpsHgx/P73N7N4cX1XG2nEOeNOZZEk9QUGbamfa7Q5405lkST1FQZtqZ9rtDnjjTiVxRF2SVJ3GLSlfq7Rvjmz0aayOMIuSeoug7akDXPG16+n7nPGG20qS6ONsDu6Lkm9h0FbUkNptKksjTTC7ui6JPUuBm1JDaXRprI00gh7o42uS5I6Z9CW1HAaaSpLI42wN9LoeqvWqSxHHnmEU1kkqQ2DtiR1opFG2BtpdB3aTmUJp7JIUhsGbUnahEYZYW+k0XVozKks3iwqqZEYtCWpl2ik0XVovKks3iwqqdEYtCWpF2mU0XVovKksjTjCLql/M2hLkrql0aayNNoIOziVRervDNqSpG7ZeCpL1n0qS6ONsDuVRZJBW5LUba1TWX7/+5vrPpWl0UbYG3EqiyPsUm2VGrQjYlpE3BcRD0bEF9o5/t2IWFjZ7o+IFVXHXqk6dl2ZdUqSej9vFu2cI+xS7Q0q68IRMRD4EfB2YClwe0Rcl5mLWttk5qer2p8NTK66xAuZOams+iRJfc/06fUdVa82ZkwRZtvbXw+djbA3Sp9JfU2ZI9oHAg9m5sOZ+RIwGzi2k/YnA1eWWI8kSTXTaFNZGm2EHZzKor4vMrOcC0ecAEzLzNMqzz8AHJSZZ7XTdiwwD9gtM1+p7GsBFgItwDcy85cdvM4MYAbAqFGjDpg9e3YZb6dfWr16NcOHD693GX2afVwb9nP57OP23XjjzvzkJ2/gqae2ZuedX+S00x7mbW97qtvX25J+Pumkg3nyySGv2T9q1Dpmz57X7Zq668Ybd+bCC9/Miy8O3LBv661f4TOfuW+L+mhL+Vmujd7ez1OnTr0jM6dsql2ZQftE4B1tgvaBmXl2O20/TxGyz67at0tmLouINwC/B/4+Mx/q7DWnTJmS8+fP79H30Z81NzfT1NRU7zL6NPu4Nuzn8tnHtbEl/dw6R7t6+sjQofWbxz5uXPtTa8aOLW6wrbVZs4ppNI8+mowZE8yc6ZSaMvX23xkR0aWgXebUkaXA7lXPdwOWddD2JNpMG8nMZZWfDwPNbDx/W5IkbQZvFu3YxjeKhjeKqseUGbRvB/aMiPERsRVFmH7N6iER8WZgJDC3at/IiNi68nhH4FBgUdtzJUlS1/nNou1zKUaVpbSgnZktwFnAb4B7gasy856I+FpEvKeq6cnA7Nx4DstewPyIuBO4iWKOtkFbkqQ+opFuFm2k0XVwKca+pNR1tDNzTma+KTP3yMyZlX3nZeZ1VW2+kplfaHPeHzNzQmbuV/n50zLrlCRJtdVIU1kaaXQdHGHvS/xmSEmSVBeNMpWlkUbXwRH2vsSgLUmS+rWNR9ez7jeKOsLedxi0JUlSv9c6uv77399c9xtFHWHftN4ylcWgLUmS1EAaaf46NN4Ie2+aymLQliRJajCNMn8dGm+EvTdNZTFoS5IkqUONNsLeiFNZOjKo3gVIkiSpsU2f3jhfST9mTDFdpL39jcYRbUmSJPUajTaVpTMGbUmSJPUajTaVpTNOHZEkSVKv0khTWTrjiLYkSZJUAoO2JEmSVAKDtiRJklQCg7YkSZJUAoO2JEmSVAKDtiRJklQCg7YkSZJUAoO2JEmSVAKDtiRJklQCg7YkSZJUAoO2JEmSVAKDtiRJklSCyMx619BjIuJpYEm96+hDdgSeqXcRfZx9XBv2c/ns49qwn8tnH9dGb+/nsZm506Ya9amgrZ4VEfMzc0q96+jL7OPasJ/LZx/Xhv1cPvu4NvpLPzt1RJIkSSqBQVuSJEkqgUFbnbms3gX0A/ZxbdjP5bOPa8N+Lp99XBv9op+doy1JkiSVwBFtSZIkqQQGbUmSJKkEBu1+LCJ2j4ibIuLeiLgnIj7ZTpumiFgZEQsr23n1qLW3i4jFEfGXSh/Ob+d4RMRFEfFgRNwVEfvXo87eLCLeXPU5XRgRz0fEp9q08fO8mSLi8oh4KiLurtq3fUTcEBEPVH6O7ODcUyptHoiIU2pXde/TQT9/KyL+WvmdcG1EbNfBuZ3+flGhgz7+SkT8rep3wjEdnDstIu6r/I7+Qu2q7n066Of/qOrjxRGxsINz+9xn2Tna/VhEjAZGZ+aCiBgB3AEcl5mLqto0AZ/JzHfVqcw+ISIWA1Mys93F+Su/3M8GjgEOAr6fmQfVrsK+JSIGAn8DDsrMJVX7m/DzvFki4nBgNfDzzNy3su//As9m5jcqoWNkZn6+zXnbA/OBKUBS/H45IDOfq+kb6CU66OejgN9nZktEfBOgbT9X2i2mk98vKnTQx18BVmfmhZ2cNxC4H3g7sBS4HTi5+v+VelV7/dzm+LeBlZn5tXaOLaaPfZYd0e7HMvPxzFxQebwKuBfYtb5V9VvHUvxSysycB2xX+YuQuufvgYeqQ7a6JzNvAZ5ts/tY4P9VHv8/4Lh2Tn0HcENmPlsJ1zcA00ortJdrr58z87eZ2VJ5Og/YreaF9SEdfJa74kDgwcx8ODNfAmZT/DegdnTWzxERwPuAK2taVB0ZtAVARIwDJgO3tXP4kIi4MyJ+HRH71LSwviOB30bEHRExo53juwKPVT1fin/p2RIn0fEvcj/PW25UZj4OxV/YgZ3baeNnumd9GPh1B8c29ftFnTurMj3n8g6mQflZ7jlvBZ7MzAc6ON7nPssGbRERw4FfAJ/KzOfbHF4AjM3M/YAfAL+sdX19xKGZuT9wNHBm5Z/WqkU75zivqxsiYivgPcB/tnPYz3Pt+JnuIRFxLtACzOqgyaZ+v6hjlwB7AJOAx4Fvt9PGz3LPOZnOR7P73GfZoN3PRcRgipA9KzOvaXs8M5/PzNWVx3OAwRGxY43L7PUyc1nl51PAtRT/FFltKbB71fPdgGW1qa7PORpYkJlPtj3g57nHPNk6tany86l22viZ7gGVm0jfBUzPDm6q6sLvF3UgM5/MzFcycz3wY9rvOz/LPSAiBgHHA//RUZu++Fk2aPdjlblSPwXuzczvdNDm9ZV2RMSBFJ+Z5bWrsveLiGGVm02JiGHAUcDdbZpdB3ywWHwkDqa4UeTxGpfaV3Q4YuLnucdcB7SuInIK8F/ttPkNcFREjKz8c/xRlX3qooiYBnweeE9mru2gTVd+v6gDbe6F+Qfa77vbgT0jYnzlX8xOovhvQJvnbcBfM3Npewf76md5UL0LUF0dCnwA+EvVUjtfAsYAZOalwAnAGRHRArwAnNTRqIo6NAq4tpLvBgFXZOZ/R8TpsKGf51CsOPIgsBY4tU619moRMZRiZYCPVe2r7mc/z5spIq4EmoAdI2IpcD7wDeCqiPgI8ChwYqXtFOD0zDwtM5+NiH+hCCkAX8vM7tyI1i900M9fBLYGbqj8/piXmadHxC7ATzLzGDr4/VKHt9DwOujjpoiYRDEVZDGV3x3VfVxZ9eUsir8oDgQuz8x76vAWeoX2+jkzf0o79870h8+yy/tJkiRJJXDqiCRJklQCg7YkSZJUAoO2JEmSVAKDtiRJklQCg7YkSZJUAoO2JPUCEdFcWT6v7Nf5RETcGxEdfQthWa/7lYj4TC1fU5LK5jraktTHRcSgzGzpYvOPA0dn5iNl1iRJ/YEj2pLUQyJiXGU0+McRcU9E/DYitqkc2zAiHRE7RsTiyuMPRcQvI+L/i4hHIuKsiPiniPhzRMyLiO2rXuIfI+KPEXF35ZstW79N7fKIuL1yzrFV1/3PiPj/gN+2U+s/Va5zd0R8qrLvUuANwHUR8ek27QdGxLcqr3NXRLR+sUdTRNwSEddGxKKIuDQiBlSOnRwRf6m8xjerrjUtIhZExJ0R8buql9m70k8PR8Qnqt7f9ZW2d0fE/96SPyNJqiVHtCWpZ+0JnJyZH42Iq4D3Av++iXP2BSYDQyi+HfTzmTk5Ir4LfPD/b+/eQay6wiiO/xeJaCJqSGkhsQkiOKCiEtFJpoiIERkfIBY2CoKFXZpgFwsVp7BLoQQLw0AKJSSRjIrG8YUDPtBKCx+NhQZkEIWJOivF2YPXA/fMJHh1IOtXnbPPfnz3FJfv7vvBBg6WftNtL5fUDfxYxu0GztjeJukTYEjS6dL/C6CrfiKjpMVUp48uAwRckXSunDq4Guix/Vctxu3AsO0lkqYCFyWNJfBLgfnAA+APYIOkS8B+YDHwBDgpqRe4CBwCum3fq/2QmAf0ADOA25J+AFYDD21/U2KfNc67jIiYNJJoR0S8Xfds3yjXV4HPJjDmrO2nwFNJw8Cvpf0W0NXSrx/A9qCkmSWxXgWsa6lvngbMKden2hx7vgI4bvsZgKRjwErgekOMq4AuSZvK/SyqHxV/A0O275a5+sv8L4A/bT8u7T8B3cArYHCsNKUW3++2R4ARSY+ojmS+BfSVHfHfbJ9viDEiYlJJoh0R8XaNtFy/Aj4q1y95Xa43rWHMaMv9KG9+T7s2zlQ70htt3259IGkZ8KxNjGoXfAMBu2wP1Nb5qiGudvPU+4+pv7sPbd8pO/BrgL2STtr+/t8GHxHxPqRGOyLi3bhPVUYBsKmhX5PNAJJWUJVxDAMDwC5JKs8WTmCeQaBX0seSpgPrgfF2igeAnZKmlHU+L2MBlkqaW2qzNwMXgCvAl6Ue/QNgC3AOuFza55Z5Pq0v1ErSbOC57aNAH7BoAp8vImJSyI52RMS70Qf8LGkrcOY/zvGk1D7PBLaVtj1UNdw3S7J9H1jbNInta5KOAEOl6bDtprIRgMNUZTDXyjqPgd7y7DKwD1hAlcQftz0q6TvgLNUu9gnbvwBI2gEcK4n5I+DrhnUXAAckjVKVo+wcJ86IiElDdrt/8CIiIpqV0pFvbTcm9xER/0cpHYmIiIiI6IDsaEdEREREdEB2tCMiIiIiOiCJdkREREREByTRjoiIiIjogCTaEREREREdkEQ7IiIiIqID/gFwpS0eFGe6GQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x504 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_learning_curve(history, 'rmse')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GMF model testing\n",
    "And finally, make a prediction and check the testing error using out-of-sample data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define rmse function\n",
    "rmse = lambda true, pred: np.sqrt(np.mean(np.square(np.squeeze(predictions) - np.squeeze(df_test.rating.values))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The out-of-sample RMSE of rating predictions is 0.9237\n"
     ]
    }
   ],
   "source": [
    "# load best model\n",
    "GMF_model = get_GMF_model(num_users, num_items, 10, 0, 0)\n",
    "GMF_model = load_trained_model(GMF_model, os.path.join(data_path, 'tmp/model.hdf5'))\n",
    "# make prediction using test data\n",
    "predictions = GMF_model.predict([df_test.userId.values, df_test.movieId.values])\n",
    "# get the RMSE\n",
    "error = rmse(df_test.rating.values, predictions)\n",
    "print('The out-of-sample RMSE of rating predictions is', round(error, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Train Multi-Layer Perceptron Model and Test Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### define MLP model architeture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_MLP_model(num_users, num_items, layers, reg_layers):\n",
    "    \"\"\"\n",
    "    Build Multi-Layer Perceptron Model Topology\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    num_users: int, total number of users\n",
    "    num_iterms: int, total number of items\n",
    "    layers: list of int, each element is the number of hidden units for each layer,\n",
    "        with the exception of first element. First element is the sum of dims of\n",
    "        user latent vector and item latent vector\n",
    "    reg_layers: list of int, each element is the L2 regularization parameter for\n",
    "        each layer in MLP\n",
    "\n",
    "    Return\n",
    "    ------\n",
    "    A Keras Model with MLP model architeture\n",
    "    \"\"\"\n",
    "    assert len(layers) == len(reg_layers)\n",
    "    num_layer = len(layers) # Number of layers in the MLP\n",
    "    # Input variables\n",
    "    user_input = Input(shape=(1,), dtype='int32', name='user_input')\n",
    "    item_input = Input(shape=(1,), dtype='int32', name='item_input')\n",
    "\n",
    "    MLP_Embedding_User = Embedding(\n",
    "        input_dim=num_users + 1,\n",
    "        output_dim=layers[0] // 2,\n",
    "        embeddings_initializer='uniform',\n",
    "        name='user_embedding',\n",
    "        embeddings_regularizer=l2(reg_layers[0]),\n",
    "        input_length=1)\n",
    "    MLP_Embedding_Item = Embedding(\n",
    "        input_dim=num_items + 1,\n",
    "        output_dim=layers[0] // 2,\n",
    "        embeddings_initializer='uniform',\n",
    "        name='item_embedding',\n",
    "        embeddings_regularizer=l2(reg_layers[0]),\n",
    "        input_length=1) \n",
    "    \n",
    "    # Crucial to flatten an embedding vector!\n",
    "    user_latent = Flatten()(MLP_Embedding_User(user_input))\n",
    "    item_latent = Flatten()(MLP_Embedding_Item(item_input))\n",
    "\n",
    "    # The 0-th layer is the concatenation of embedding layers\n",
    "    vector = Concatenate(axis=-1)([user_latent, item_latent])\n",
    "\n",
    "    # MLP layers\n",
    "    for idx in range(1, num_layer):\n",
    "        layer = Dense(\n",
    "            units=layers[idx],\n",
    "            activation='relu',\n",
    "            kernel_initializer='glorot_uniform',\n",
    "            kernel_regularizer=l2(reg_layers[idx]),\n",
    "            name = 'layer%d' %idx)\n",
    "        vector = layer(vector)\n",
    "    \n",
    "    # Final prediction layer\n",
    "    prediction = Dense(1, kernel_initializer='glorot_uniform', name='prediction')(vector)\n",
    "    \n",
    "    # Stitch input and output\n",
    "    model = Model([user_input, item_input], prediction)\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### create MLP model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "user_input (InputLayer)         (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "item_input (InputLayer)         (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "user_embedding (Embedding)      (None, 1, 32)        21504       user_input[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "item_embedding (Embedding)      (None, 1, 32)        290144      item_input[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "flatten_4 (Flatten)             (None, 32)           0           user_embedding[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "flatten_5 (Flatten)             (None, 32)           0           item_embedding[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "concatenate (Concatenate)       (None, 64)           0           flatten_4[0][0]                  \n",
      "                                                                 flatten_5[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "layer1 (Dense)                  (None, 32)           2080        concatenate[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "layer2 (Dense)                  (None, 16)           528         layer1[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "layer3 (Dense)                  (None, 8)            136         layer2[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "prediction (Dense)              (None, 1)            9           layer3[0][0]                     \n",
      "==================================================================================================\n",
      "Total params: 314,401\n",
      "Trainable params: 314,401\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "MLP_model = get_MLP_model(num_users, num_items, [64, 32, 16, 8], [0, 0, 0, 0])\n",
    "MLP_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### train MLP model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Kevin/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:108: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60002 samples, validate on 20001 samples\n",
      "Epoch 1/30\n",
      "60002/60002 [==============================] - 6s 105us/step - loss: 1.1327 - mean_squared_error: 1.1327 - rmse: 1.0321 - val_loss: 0.8792 - val_mean_squared_error: 0.8792 - val_rmse: 0.9327\n",
      "Epoch 2/30\n",
      "60002/60002 [==============================] - 4s 74us/step - loss: 0.8239 - mean_squared_error: 0.8239 - rmse: 0.9028 - val_loss: 0.8250 - val_mean_squared_error: 0.8250 - val_rmse: 0.9032\n",
      "Epoch 3/30\n",
      "60002/60002 [==============================] - 5s 77us/step - loss: 0.7563 - mean_squared_error: 0.7563 - rmse: 0.8648 - val_loss: 0.8108 - val_mean_squared_error: 0.8108 - val_rmse: 0.8958\n",
      "Epoch 4/30\n",
      "60002/60002 [==============================] - 4s 69us/step - loss: 0.7155 - mean_squared_error: 0.7155 - rmse: 0.8413 - val_loss: 0.8267 - val_mean_squared_error: 0.8267 - val_rmse: 0.9037\n",
      "Epoch 5/30\n",
      "60002/60002 [==============================] - 4s 68us/step - loss: 0.6923 - mean_squared_error: 0.6923 - rmse: 0.8268 - val_loss: 0.8318 - val_mean_squared_error: 0.8318 - val_rmse: 0.9064\n",
      "Epoch 6/30\n",
      "60002/60002 [==============================] - 4s 68us/step - loss: 0.6691 - mean_squared_error: 0.6691 - rmse: 0.8132 - val_loss: 0.8099 - val_mean_squared_error: 0.8099 - val_rmse: 0.8950\n",
      "Epoch 7/30\n",
      "60002/60002 [==============================] - 4s 69us/step - loss: 0.6480 - mean_squared_error: 0.6480 - rmse: 0.8002 - val_loss: 0.8273 - val_mean_squared_error: 0.8273 - val_rmse: 0.9042\n",
      "Epoch 8/30\n",
      "60002/60002 [==============================] - 4s 69us/step - loss: 0.6268 - mean_squared_error: 0.6268 - rmse: 0.7867 - val_loss: 0.8241 - val_mean_squared_error: 0.8241 - val_rmse: 0.9027\n",
      "Epoch 9/30\n",
      "60002/60002 [==============================] - 4s 67us/step - loss: 0.6013 - mean_squared_error: 0.6013 - rmse: 0.7703 - val_loss: 0.8525 - val_mean_squared_error: 0.8525 - val_rmse: 0.9176\n",
      "Epoch 10/30\n",
      "60002/60002 [==============================] - 4s 68us/step - loss: 0.5763 - mean_squared_error: 0.5763 - rmse: 0.7544 - val_loss: 0.8585 - val_mean_squared_error: 0.8585 - val_rmse: 0.9219\n",
      "Epoch 11/30\n",
      "60002/60002 [==============================] - 4s 68us/step - loss: 0.5525 - mean_squared_error: 0.5525 - rmse: 0.7383 - val_loss: 0.8671 - val_mean_squared_error: 0.8671 - val_rmse: 0.9259\n",
      "Epoch 12/30\n",
      "60002/60002 [==============================] - 4s 67us/step - loss: 0.5291 - mean_squared_error: 0.5291 - rmse: 0.7224 - val_loss: 0.8945 - val_mean_squared_error: 0.8945 - val_rmse: 0.9400\n",
      "Epoch 13/30\n",
      "60002/60002 [==============================] - 4s 67us/step - loss: 0.5041 - mean_squared_error: 0.5041 - rmse: 0.7053 - val_loss: 0.9037 - val_mean_squared_error: 0.9037 - val_rmse: 0.9450\n",
      "Epoch 14/30\n",
      "60002/60002 [==============================] - 4s 67us/step - loss: 0.4803 - mean_squared_error: 0.4803 - rmse: 0.6883 - val_loss: 0.9160 - val_mean_squared_error: 0.9160 - val_rmse: 0.9516\n",
      "Epoch 15/30\n",
      "60002/60002 [==============================] - 4s 67us/step - loss: 0.4592 - mean_squared_error: 0.4592 - rmse: 0.6724 - val_loss: 0.9594 - val_mean_squared_error: 0.9594 - val_rmse: 0.9737\n",
      "Epoch 16/30\n",
      "60002/60002 [==============================] - 4s 68us/step - loss: 0.4387 - mean_squared_error: 0.4387 - rmse: 0.6574 - val_loss: 0.9569 - val_mean_squared_error: 0.9569 - val_rmse: 0.9728\n",
      "Epoch 00016: early stopping\n"
     ]
    }
   ],
   "source": [
    "# model config\n",
    "BATCH_SIZE = 64\n",
    "EPOCHS = 30\n",
    "VAL_SPLIT = 0.25\n",
    "\n",
    "# train model\n",
    "history = train_model(MLP_model, 'adam', BATCH_SIZE, EPOCHS, VAL_SPLIT, \n",
    "                      inputs=[df_train.userId.values, df_train.movieId.values],\n",
    "                      outputs=df_train.rating.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MLP learning curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtoAAAG5CAYAAACwZpNaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xl83VWd//HXp6FQ0hZatlJokxREli62JWzD1rIWBmUVwQyKClUG1MGfClIEBq0yCqgMqFNH3CbSYdhkRtwQoqIoUCxb2aEbe1lKS1qg7fn9cW6atE3btDff3iR9PR+P+7j3e77n+73nnnR559zzPd9IKSFJkiSpc/WqdAMkSZKknsigLUmSJBXAoC1JkiQVwKAtSZIkFcCgLUmSJBXAoC1JkiQVwKAtSZ0oIuoiIkXEJh2oe3pE3LUh2lWOiFgYETtVuh2S1N0YtCVttCJiZkS8ExHbrFQ+vRSW6yrTsnUL7EVLKfVLKT1TxLkj4r0R8T8RMS8i5kfEgxHxuYioKuL9JGlDMmhL2tg9C5zashERI4HNK9ecDauSgTYidgb+BswBRqaUtgQ+CNQD/dfjfBX/pUSS2jJoS9rY/Qz4SJvtjwI/bVshIraMiJ9GxCsRMSsiLoyIXqV9VRFxeWlE9hngH9s59ocR8UJEPBcRXy033EZEr4g4PyKejohXI+L6iNiqzf7/iYgXSyPEf4yI4W32/TgivhcRt0XEW8D4Utk1EfHLiFgQEX8rheCWY1JEvKfN8Wuqe0REPF567+9GxB8i4ozVfJR/Bf6SUvpcSukFgJTS4ymlD6eU3oiIcRExd6XPPjMiDiu9viQiboiI/4qIN4ELImLRSn0xpvSz6V3a/nhEPBoRr0fEbyKidv1/EpK0ZgZtSRu7vwJbRMTupQD8IeC/Vqrz78CWwE7AweRg/rHSvjOBY4Ax5JHYk1Y69ifAEuA9pTpHAKsLnh31GeC4Ult2AF4Hrmmz/1fALsB2wP1A40rHfxiYTB41bpkjfio5+A4EnirtX51265am4NwAfAnYGngc+Ic1nOewUv1yHFs6xwDgm8DdwIlt9n8YuCGl9G5EHAdcAJwAbAv8CbiuzPeXpNUyaEtS66j24cBjwHMtO9qE7y+llBaklGYCVwCnlaqcDHw7pTQnpfQa8PU2xw4CjgL+JaX0VkrpZeBbwClltveTwKSU0tyU0tvAJcBJLVMnUkrXltrasu99EbFlm+N/kVL6c0ppWUppcansppTSPSmlJeRgPnoN77+6ukcDj6SUbirtuwp4cQ3n2Rp4YV0+eDvuTindUvosi4CfU5oKFBFB7uufl+p+Evh6SunRUvu+Box2VFtSUZzPJkk5aP8RGMZK00aAbYBNgVltymYBO5Ze70CeY9x2X4taoDfwQs58QB7gaFt/fdQCN0fEsjZlS4FBEfEieYT5g+RR25Y62wDzS6/be/+2gbgZ6LeG919d3RX6IqWUVp76sZJXgcFr2N8RK3+WG4B/j4gdyKP6iTxyDbnfvhMRV7SpH+Sf5SwkqZM5oi1po5dSmkW+KPJo4KaVds8D3iWHtBY1tI56vwAMXWlfiznA28A2KaUBpccWKaXhlGcOcFSbcw5IKfVJKT1HnipxLHlaxpZAXemYaHN8KvP9V+cFYEjLRmlEecjqq3M7K07zWNlbQHWb81WRf3loa4XPklJ6A/gt+ZuGDwPXpZRa6swBPrlSv22eUvrLmj+WJK0fg7YkZZ8ADkkpvdW2MKW0FLgemBwR/UvTDD5H6zzu64HPRMSQiBgInN/m2BfIoe+KiNiidBHjzhFx8Dq0a7OI6NPm0Qv4fqk9tQARsW1EHFuq358c7l8lh9SvrVs3lOWXwMiIOK40jeVsYPs11L8Y+IeI+GZEbA8QEe8pXdw4AHgC6BMR/1i6mPFCYLMOtOPn5KlAJ9I6bQRyv32p5eLQ0oWqH1zHzyhJHWbQliQgpfR0Sum+1ez+NHl09RnyxYM/B64t7fsB8BvgAfKFhyuPiH+EPPVkBvmixRtYt+kSC4FFbR6HAN8BbgV+GxELyBd07lOq/1PyNIjnSu/513V4r7KklOaRp6x8gxz09wDuIwf/9uo/DexHHnV/JCLmAzeWjlmQUpoP/DPwn+TP8xawpqkoLW4lTxt5KaX0QJv3uxn4N2BqaZWSh8lz6CWpENH6jZokSZ2nNPo+F2hIKd1Z6fZI0obmiLYkqdNExJERMSAiNiMvpRdswFF1SepKDNqSpM60H/A0+SLS9wPHlZbdk6SNTmFTRyLiWvJNHF5OKY1oZ3+Q5xkeTV4e6vSU0v2lfUuBh0pVZ6eUPlBIIyVJkqSCFDmi/WNgwhr2H0W+WGUXYCLwvTb7FqWURpcehmxJkiR1O4XdsCal9MeIqFtDlWOBn5bWN/1raU7f4NJyWOtsm222SXV1a3q7jddbb71F3759K92Mbs9+LJ99WD77sHz2Yfnsw/LZh52jUv04bdq0eSmlldf1X0Ul7wy5Iyve0WtuqewF8rqp9wFLgMtSSre0d4KImEgeDWfQoEFcfvnlxba4m1q4cCH9+q3pJm/qCPuxfPZh+ezD8tmH5bMPy2cfdo5K9eP48eM7dDfZSgbtaKesZcJ4TUrp+YjYCbgjIh4qrbe6YuWUpgBTAOrr69O4ceMKa2x31tTUhH1TPvuxfPZh+ezD8tmH5bMPy2cfdo6u3o+VXHVkLivetngI8DxASqnl+RmgCRizoRsnSZIklaOSQftW4COR7QvMTym9EBEDS+uvEhHbAPuT724mSZIkdRuFTR2JiOuAccA2ETEXuBjoDZBS+j5wG3lpv6fIy/t9rHTo7sB/RMQy8i8Cl6WUDNqSJKlbeffdd5k7dy6LFy9eZd+WW27Jo48+WoFW9SxF92OfPn0YMmQIvXv3Xq/ji1x15NS17E/A2e2U/wUYWVS7JEmSNoS5c+fSv39/6urqyLcPabVgwQL69+9foZb1HEX2Y0qJV199lblz5zJs2LD1Ood3hpQkSSrA4sWL2XrrrVcJ2eoeIoKtt9663W8kOsqgLUmSVBBDdvdW7s/PoC1JkiQVwKAtSZLUA73xxht897vfXa9jjz76aN5444011rnooou4/fbb1+v8GwuDtiRJUhfQ2Ah1ddCrV35ubCzvfGsK2kuXLl3jsbfddhsDBgxYY51LL72Uww47bL3btzpra1t3YtCWJEmqsMZGmDgRZs2ClPLzxInlhe3zzz+fp59+mtGjR/OFL3yBpqYmxo8fz4c//GFGjswLvB133HHsueeeDB8+nClTpiw/tq6ujnnz5jFz5kx23313zjzzTIYPH84RRxzBokWLADj99NO54YYblte/+OKLGTt2LCNHjuSxxx4D4JVXXuHwww9n7NixfPKTn6S2tpZ58+at0tZ+/fpx0UUXsc8++3D33XdTV1fHBRdcwH777Ud9fT33338/Rx55JDvvvDPf//73AXjhhReYMGECo0ePZsSIEfzpT38C4Le//S377bcfY8eO5YMf/CALFy5c/04sk0FbkiSpwiZNgubmFcuam3P5+rrsssvYeeedmT59Ot/85jcBuOeee5g8eTIzZuRblFx77bVMmzaN++67j6uuuopXX311lfM8+eSTnH322TzyyCMMGDCAG2+8sd3322abbbj//vs566yzuPzyywH413/9Vw455BDuv/9+jj/+eGbPnt3usW+99RYjRozgb3/7GwcccAAAQ4cO5e677+bAAw9cHur/+te/ctFFFwHw85//nEMPPZTp06fzwAMPMHr0aObNm8dXv/pVbr/9du6//37q6+u58sor178Ty1TYOtqSJEnqmNXkz9WWr6+99957hTWhr7rqKm6++WYA5syZw5NPPsnWW2+9wjHDhg1j9OjRAOy5557MnDmz3XOfcMIJy+vcdNNNANx1113Lzz9hwgQGDhzY7rFVVVWceOKJK5R94AMfAGDkyJEsXLiQ/v37079/f/r06cMbb7zBXnvtxemnn06vXr047rjjGD16NH/4wx+YMWMG+++/PwDvvPMO++23X4f7p7M5ol2Gzp5LJUmSNk41NetWvr769u27/HVTUxO33347d999Nw888ABjxoxpd83ozTbbbPnrqqoqlixZ0u65W+q1rZPvT7h2ffr0oaqqqt3z9erVa4U29OrViyVLlnDQQQfx61//mh133JHTTjuNn/70p6SUOPzww5k+fTrTp09nxowZ/PCHP+xQG4pg0F5PRcylkiRJG6fJk6G6esWy6upcvr769+/PggULVrt//vz5DBw4kOrqah577DH++te/rv+brcYBBxzA9ddfD+S506+//nqnnXvWrFlsu+22nHnmmXziE5/g/vvvZ9999+XPf/4zTz31FADNzc088cQTnfae68qgvZ6KmEslSZI2Tg0NMGUK1NZCRH6eMiWXr6+tt96a/fffnxEjRvCFL3xhlf0TJkxgyZIljBo1ii9/+cvsu+++ZXyC9l188cX89re/ZezYsfzqV79i8ODBnXbL9KamJvbff3/GjBnDjTfeyGc/+1m23XZbfvzjH3PqqacyatQo9t133+UXZlZCdHRIv6urr69P99133wZ7v1698kj2yiJg2bIN1owOaWpqYty4cZVuRrdnP5bPPiyffVg++7B89mHHPProo+y+++7t7luwYEGnBc6u7O2336aqqopNNtmEu+++m7POOovp06d32vk3RD+293OMiGkppfq1HevFkOuppiZPF2mvXJIkSTB79mxOPvlkli1bxqabbsoPfvCDSjdpgzJor6fJk/Oc7LbTR8qdSyVJktST7LLLLvz973+vdDMqxjna66mIuVSSJEnqORzRLkNDg8FakiRJ7XNEW5IkSSqAQVuSJEkqgEFbkiRJAPTr1w+A559/npNOOqndOuPGjWNtSyp/+9vfprnNihFHH300b7zxRuc1tJswaEuSJGkFO+ywAzfccMN6H79y0L7tttsYMGBAZzRtudXdCr4rMWhLkiT1QOeddx7f/e53l29fcsklXHHFFSxcuJBDDz2UsWPHMnLkSH7xi1+scuzMmTMZMWIEAIsWLeKUU05h1KhRfOhDH2LRokXL65111lnU19czfPhwLr74YgCuuuoqnn/+ecaPH8/48eMBqKurY968eQBceeWVjBgxghEjRvDtb397+fvtvvvunHnmmQwfPpwjjjhihfdpcfrpp/O5z32O8ePHc9555/G1r32Nj370oxxxxBHU1dVx00038cUvfpGRI0cyYcIE3n33XQDOP/989thjD0aNGsXnP/95AF555RVOPPFE9tprL/baay/+/Oc/l93nK3PVEUmSpIL9y79A2xsiLl26OVVV5Z1z9Ggo5dR2nXLKKfzLv/wL//zP/wzA9ddfz69//Wv69OnDzTffzBZbbMG8efPYd999+cAHPkBEtHue733ve1RXV/Pggw/y4IMPMnbs2OX7Jk+ezFZbbcXSpUs59NBDefDBB/nMZz7DlVdeyZ133sk222yzwrmmTZvGj370I/72t7+RUmKfffbh4IMPZuDAgTz55JNcd911/OAHP+Dkk0/mxhtv5J/+6Z9Wac8TTzzB7bffTlVVFV/60pd4+umnufPOO5kxYwb77bcfN954I9/4xjc4/vjj+eUvf8lBBx3EzTffzGOPPUZELJ/C8tnPfpZzzz2XAw44gNmzZ3PkkUfy6KOPruuPYY0c0ZYkSeqBxowZw8svv8zzzz/PAw88wMCBA6mpqSGlxAUXXMCoUaM47LDDeO6553jppZdWe54//vGPywPvqFGjGDVq1PJ9119/PWPHjmXMmDE88sgjzJgxY41tuuuuuzj++OPp27cv/fr144QTTuBPf/oTAMOGDWP06NEA7LnnnsycObPdc3zwgx+kqs1vKUcddRS9e/dm5MiRLF26lAkTJgAwcuRIZs6cyRZbbEGfPn0444wzuOmmm6iurgbg9ttv55xzzmH06NF84AMf4M0332TBggVr6dV144i2JElSwVYeeV6wYBH9+/cv/H1POukkbrjhBl588UVOOeUUABobG3nllVeYNm0avXv3pq6ujsWLF6/xPO2Ndj/77LNcfvnl3HvvvQwcOJDTTz99redJKa1232abbbb8dVVVVbtTRwD69u3b7nG9evWid+/ey9vaq1cvlixZwiabbMI999zD73//e6ZOncrVV1/NHXfcwbJly7j77rvZfPPN19jmcjiiLUmS1EOdcsopTJ06lRtuuGH5KiLz589nu+22o3fv3tx5553MmjVrjec46KCDaGxsBODhhx/mwQcfBODNN9+kb9++bLnllrz00kv86le/Wn5M//792x0dPuigg7jllltobm7mrbfe4uabb+bAAw/srI/broULFzJ//nyOPvpovv3tbzO9NIfniCOO4Oqrr15eb3rbuT2dxBFtSZKkHmr48OEsWLCAHXfckcGDBwPQ0NDA+9//furr6xk9ejS77bbbGs9x1lln8bGPfYxRo0YxevRo9t57bwDe9773MWbMGIYPH85OO+3E/vvvv/yYiRMnctRRRzF48GDuvPPO5eVjx47l9NNPX36OM844gzFjxqx2mkhnWLBgAcceeyyLFy8mpcS3vvUtIF+0efbZZzNq1CiWLFnCQQcdxPe///1Ofe9Y0xB+d1JfX5/WtqbjxqqpqYlx48ZVuhndnv1YPvuwfPZh+ezD8tmHHfPoo4+y++67t7tvwYIFG2TqSE+3IfqxvZ9jRExLKdWv7VinjkiSJEkFMGhLkiRJBTBoS5IkFaSnTNHdWJX78zNoS5IkFaBPnz68+uqrhu1uKqXEq6++Sp8+fdb7HK46IkmSVIAhQ4Ywd+5cXnnllVX2LV68uKwAp6zofuzTpw9DhgxZ7+MN2pIkSQXo3bs3w4YNa3dfU1MTY8aM2cAt6nm6ej86dUSSJEkqgEFbkiRJKoBBW5IkSSqAQVuSJEkqQGFBOyKujYiXI+Lh1eyPiLgqIp6KiAcjYmybfR+NiCdLj48W1UZJkiSpKEWOaP8YmLCG/UcBu5QeE4HvAUTEVsDFwD7A3sDFETGwwHZKkiRJna6woJ1S+iPw2hqqHAv8NGV/BQZExGDgSOB3KaXXUkqvA79jzYFdkiRJ6nIquY72jsCcNttzS2WrK19FREwkj4YzaNAgmpqaCmlod7dw4UL7phPYj+WzD8tnH5bPPiyffVg++7BzdPV+rGTQjnbK0hrKVy1MaQowBaC+vj6NGzeu0xrXkzQ1NWHflM9+LJ99WD77sHz2Yfnsw/LZh52jq/djJVcdmQsMbbM9BHh+DeWSJElSt1HJoH0r8JHS6iP7AvNTSi8AvwGOiIiBpYsgjyiVSZIkSd1GYVNHIuI6YBywTUTMJa8k0hsgpfR94DbgaOApoBn4WGnfaxHxFeDe0qkuTSmt6aJKSZIkqcspLGinlE5dy/4EnL2afdcC1xbRLkmSJGlD8M6QkiRJUgEM2pIkSVIBDNqSJElSAQzakiRJUgEM2pIkSVIBDNqSJElSAQzakiRJUgEM2pIkSVIBDNqSJElSAQzakiRJUgEM2pIkSVIBDNqSJElSAQzakiRJUgEM2pIkSVIBDNqSJElSAQzakiRJUgEM2pIkSVIBNql0AyRJktS1vPoqPPkkbLop9O7d/qPtvqoqiKh0q7seg7YkSZIAePll+OY34ZprYNGidTu2vQC+pnDekfK1HdOv3+bFdEQnMWhLkiRt5NoG7LffhoYG+OAHYdkyePfdVR/vvNN++Zr2rVy+aBG8+WbHj0lp1XZfcMEWG76z1oFBW5IkaSPVErC/+11YvDgH7AsvhPe+t9ItW9XSpauG8OnTXwF2r3TTVsugLUmStJHpTgG7RVVVfvTp01rWp8+yyjWoAwzakiRJG4mXX4bLL89TRLpLwO7ODNqSJEk93MoB+8MfzgF7110r3bKezaAtSZLUQxmwK8ugLUmS1MO88koO2FdfbcCuJIO2JElSD7FywD711Bywd9ut0i3bOBm0JUmSujkDdtdk0JYkSeqmXnkFrrgiB+zm5tYpIgbsrsGgLUmS1M2sHLBPPRW+/GUDdldj0JYkSeomDNjdi0FbkiSpi5s3r3UOdkvAvvBC2L3r3n1cGLQlSZK6rHnz8gj2v/97DtinnJJHsA3Y3YNBW5IkqYsxYPcMBm1JkqQuwoDdsxi0JUmSKmzePLjyyhyw33oLPvShHLD32KPSLVM5DNqSJEkVYsDu2XoVefKImBARj0fEUxFxfjv7ayPi9xHxYEQ0RcSQNvuWRsT00uPWItspSZK0Ic2f35sLLoBhw+Cyy+CYY+Dhh+G66wzZPUlhI9oRUQVcAxwOzAXujYhbU0oz2lS7HPhpSuknEXEI8HXgtNK+RSml0UW1T5IkqWhvvw1PPQWPPw5PPJGfH38c/v73fXn7bTj55DyCPXx4pVuqIhQ5dWRv4KmU0jMAETEVOBZoG7T3AM4tvb4TuKXA9kiSJHW6lOC551YM0i3BeuZMWLaste7gwbDrrjBhwot89as7GrB7uEgpFXPiiJOACSmlM0rbpwH7pJTOaVPn58DfUkrfiYgTgBuBbVJKr0bEEmA6sAS4LKW0SgiPiInARIBBgwbtOXXq1EI+S3e3cOFC+vXrV+lmdHv2Y/nsw/LZh+WzD8u3sfZhc3MVc+Zszpw51cyZU83cua2vFy+uWl6vT5+lDB3azJAhixg6tHn5Y8iQRfTtuxTYePuws1WqH8ePHz8tpVS/tnpFjmhHO2Urp/rPA1dHxOnAH4HnyMEaoCal9HxE7ATcEREPpZSeXuFkKU0BpgDU19encePGdWLze46mpibsm/LZj+WzD8tnH5bPPixfT+7DJUvyKPTKUz0efxxeeKG1XgTU1eXR6aOPzs/vfW9+3nHHKiL6A/1X+z49uQ83pK7ej0UG7bnA0DbbQ4Dn21ZIKT0PnAAQEf2AE1NK89vsI6X0TEQ0AWOAFYK2JEnSukoJXn111Wkejz+e51O/+25r3a22yuH5iCPyc8tj552hT5/KfQZ1D0UG7XuBXSJiGHmk+hTgw20rRMQ2wGsppWXAl4BrS+UDgeaU0tulOvsD3yiwrZIkqYdZvLj9CxEffxxef721Xu/e8J735AD9gQ+0jkzvuitss03l2q/ur7CgnVJaEhHnAL8BqoBrU0qPRMSlwH0ppVuBccDXIyKRp46cXTp8d+A/ImIZeQnCy1ZarUSSJG3k3noL5sxpfcye3fr6qafyFJC2l6LtsEMO0SefvOLodG0tbOKdRVSAQv9YpZRuA25bqeyiNq9vAG5o57i/ACOLbJskSeq63nkH5s5dMUivHKjbjkq32H57GDoU9tkHTjutNUy/973Qf/VTpqVC+PubJEnaoJYuhRdfXP1o9Jw58NJLK45GQ54vPXQo1NTA/vu3vh46ND923BE23bQyn0lqj0FbkiR1mpYLDdsLzy2P557Lq3u01bdva3AeNao1PLd99O1bmc8krS+DtiRJ6rClS2HmzGp+/ev2R6TnzoVFi1Y8ZtNNYciQHJYPPHDVkeihQ2HAgLxkntSTGLQlSdIavfMO3Hkn3Hgj3HILvPLK3sv39eqV73ZYUwNjxuRVO9oG6Joa2HbbXE/a2Bi0JUnSKpqb4Te/gZtugv/9X5g/H/r1g2OOgbq6RznmmN0ZOjSv5OGKHVL7/KshSZKAHKb/7/9yuP7Vr/IUkK22ghNOgBNPhEMPzTdpaWp6if33373SzZW6PIO2JEkbsVdegV/8Iofr22/Pd0XcYQf4+MdzwD7oIEespfXlXx1JkjYyc+bAzTfncP2nP8GyZbDTTvDZz+aR6733dk611BkM2pIkbQSefDIH65tugnvuyWUjRsCFF+aR61GjXPVD6mwG7TL98If5H6e99qp0SyRJapUSPPRQXinkppvg4Ydz+V57wde/nsP1e99b2TZKPZ1Buwxvvw1f+xrMmgXnnQcXXQSbbVbpVkmSNlbLluXR6paR66efzlNADjwQvvMdOO64vNyepA3DoF2GzTaDadPg3HNz4L71VvjJT2Ds2Eq3TJK0sViyJM+zvummPO/6ueegd++8Qsh558Gxx8J221W6ldLGyaBdpgED4Ec/yhePTJyYLyCZNCk/Nt200q2TJPVEb7+dVwi56aa8Ysirr8Lmm8NRR+UpIf/4j/n/J0mVZdDuJMcck+e/ffazcOml+R++n/wE3ve+SrdMktQTLFwIv/51nnP9y1/CggWwxRbw/vfncH3kkdC3b6VbKaktg3Yn2mor+NnP4KST4JOfhPr6PG/7/PPz13iSJK2L11/Pd2W86aZ8l8bFi2GbbeBDH8rfpB5yiN+eSl2ZQbsAxx4LBxwAn/50Dtq33JJHt0eMqHTLJEld2ZIl8Pzz+a6MN94Id96Zy4YMydMTTzgh//9SVVXplkrqCIN2QbbeGn7+8zzicNZZsOeecMkl8IUveIctSdqYLFmS77740kutjxdfbH973ry8LB/Ae94D/+//5f9H6utd41rqjox8BTvxxHz72n/+Z7jggnxF+I9/DHvsUemWSZLW15IlORS3DcwdCc9tVVfDoEGw/fY5VO+/f+v2gQfC8OGGa6m7M2hvANtuC//zP3D99Tlwjx0LX/kKfO5zfv0nSV1Fe+F55QBdTngeNGjF1/36bfjPKGnDMmhvQCefDAcfDJ/6FHzxi/nilh//GHbdtdItk6SeKSV47TV44YX8ePFF+POfh/K//7tqmF5beB40aPXhuWXb8CypLYP2BjZoUA7Y110H55wDo0fnm9185jOObktSR737bg7HL764Yohe+fWLL+a6K9qZzTdvDco77wz/8A+rjji3PPr1cwqHpPVj0K6ACPjwh2H8+HwV+ec+l8P3j36UR0skaWOUUl4rek3BueX1vHntn2PbbXNQHjwYdt+99fXgwa2vn3zyTxx11IGGZ0mFM2hX0ODB+bbtP/tZHtEeNQr+7d/g7LOhV69Kt06SOseyZXnVjbWNPr/wAjQ3r3r8ppu2huSdd87L27UNzi2vBw3q2D0LnntuqSFb0gZh0K6wCPjIR+DQQ+HMM3PgvvFGuPZa2GmnSrdOkjomJbj//nzHwjlzVgzRL70ES5euesyWW7aG5b33bn8fgc1eAAAgAElEQVT0efBgGDjQqRuSuieDdhex4475P6gf/QjOPTePbn/zm/kOk45uS+qKUoIHH4T//u+8qtLTT+dAPGhQa1gePbr90eftt88XGUpST2bQ7kIi4OMfh8MOgzPOyEsB3ngj/PCHUFtb6dZJUvboozlc//d/w2OP5Qu5DzkEvvQlOP542GqrSrdQkroGg3YXVFMDv/kN/OAH+a5gI0bAlVfm8O3Xp5Iq4amnWsP1Qw/lf4sOPhg++9l8Y65tt610CyWp63FSQhcVkVckeeihPHdx4kSYMCHPfZSkDWHmTPjGN2DPPWGXXeDCC2GLLeCqq+C55+DOO/N9AQzZktQ+g3YXV1cHv/sdXHMN3HVXHt3+0Y/av6mCJJVr7lz41rdg331h2DA47zzYZBO44gqYPTv/O/TpT+e51pKkNTNodwO9euX52g89lC8s+vjH4Zhj8oiSJJXrxRfh6qvhwANh6NC8tv8778Bll8Ezz8Df/pbLhg6tdEslqXsxaHcjO+2Uv6r9znfy84gR8NOfOrotad3Nmwf/8R/5IsYdd8yj1G+8AV/5Cjz+eF6q77zz8qi2JGn9GLS7mV698lrbDzwAw4fDRz8Kxx6b16uVpDV5/fW8Rv+RR+bl9T71qfzN2KRJ8PDD+VuzCy+E97630i2VpJ7BVUe6qV12gT/8IY9uT5qUQ/fVV8Opp7oyiaRWb74Jv/hFXi3kt7+Fd9/N34598YvwoQ/lNfv9N0OSiuGIdjdWVZXnTU6fDrvuCg0NeZmtl16qdMu0oaSUb1n9/PP5FtcSwFtvwdSpeU3r7bbLd5996KG8FN+99+al+r72NXjf+wzZklQkR7R7gF13zSsBXHklfPnLeXT7u9+Fk0+udMu0NinBwoUwf36eH9veY3X7Wsrffbf1fLvvDocemufdjhuXb12tjcOiRXDbbXnk+v/+L28PHpynh3zoQ7DPPt5lVpI2NIN2D1FVBV/4AvzjP8Lpp+f/WK+4Io90PvfcwdTUwOTJedRbnWfZMliwYN3DcdvtpUvX/B7V1TBgAGy5ZX7edts8dWjAgBXL58/PF8lee22eRhQBY8fm0H3ooXDAAdC374bpl+5o2bL8rcCmm0KfPvnR1Ud7334739zqv/8bbr01/9K27bat/wYccED+t0GSVBkG7R5mjz3gL3/Jgfr661tKg1mz8k1vwLDdUa+8Ag8+mB+PPAKPPTaCTTZZMSTPn7/2VV/69WsNxQMGwA475J9TS0Be+dG2fMstc/DrqC9+MS/Lds89cMcd8Pvfw7e/Dd/8JvTunddGbgne++yzbufuaZ5/Pk+juOee/LjvvvxzbatPH9h88/xo+7rI7bWNOi9ZEvzqVzlc33JL/jO41VZwyik5XI8bl9e9liRVXqH/HEfEBOA7QBXwnymly1baXwtcC2wLvAb8U0ppbmnfR4ELS1W/mlL6SZFt7Uk22SSve7uy5mY480x49NE8vWT48DztZLPNNnwbu5J33sl90hKqWx4vvthaZ7vtoH//PgwZArW1eW7rmgJyy2OLLTZ86Nl00zySecABcNFF+ed+112twfsrX4F//dc8Un7AAa1TTcaM6bmjn2+8kYN0S7C+997WdeirqmDkyDzVavjwPLK9aFF+LF7c+nrl7TffhJdfbn//2r6lWJNNN119EO/TB+677x9488385+2443K4Puyw/IuUJKlrKSwCREQVcA1wODAXuDcibk0pzWhT7XLgpymln0TEIcDXgdMiYivgYqAeSMC00rGvF9Xenmb27PbLFy3KN6FoCQJVVXkaQkvwHjEiP++yS8/7jzulvAziyoH60UdhyZJcZ7PN8uefMCGvxjBqVA5h220HTU33MW7cuIp+hvVRXQ1HHJEfkJd4++Mfc+i+4468VjLkXwzGjWsd8d59964/daI9ixfnC4TbjlY/8UTr/l12gYMPhr33hr32yjeBqq7u3Da8++6aQ3o523vv/Rqf/vQgjjzSX5Ilqasrcqxtb+CplNIzABExFTgWaBu09wDOLb2+E7il9PpI4HcppddKx/4OmABcV2B7e5SaGpg1a9Xy2tp8M4onnsjTIR5+OD8/+CDcdFPrNIjevfNod0sAbwnhO+/cPUY9Fy2CGTNaw/QDD+TnV19trTN0aA7SxxyTn9/3vhzCevrX7gMH5rXXjz02b7/4Yp7b3RK8byn9Ldx++xy6W4J3XV3FmrxaS5fmX5RaRqnvuSf/nFt+cdp++zxF5iMfycG6vn7DXCDau3d+9O/f+eduanqUceMGdf6JJUmdrshIsSMwp832XGCfleo8AJxInl5yPNA/IrZezbE7FtfUnmfy5Dwnu7m5tay6OpdvtlkepR05csVjFi2Cxx5bMYDfc0+eC9pis81gt91aR75bAnhdXWVWNEgJ5sxpDdItjyeeyFMAIH/ukSPhhBNWHKV2RY5s++3z+uunnpq3n302B+6Wx89/nsuHDWsN3ePH5+M2pJRg5swVp39Mm5aXsoM8TWevveDzn28drd5xx+45Ki9J6hkiFXT/7oj4IHBkSumM0vZpwN4ppU+3qbMDcDUwDPgjOXQPByYCm6WUvlqq92WgOaV0xUrvMbFUl0GDBu05derUQj5Ld3X77dvxn/+5Ey+/vBnbbfc2Z5zxDIcd9vI6n2fRoipmzapm5sy+zJxZzbPP9mXmzL68/HKf5XX69FlKTU0zw4a9RV1dfgwb9hbbbfd2pwWdRYuqePbZvjz9dF+eeaYfzzzTl6ef7sdbb7X+vrjDDovYaaeF7LTTW+y000J23vktBg9e1Cmj8AsXLqRfv37ln6gbSQlmzarm/vsH8ve/D2D69AEsXJjnFNXVvcWYMa8zduwbjB79Bv36LVnr+dalD994ozePPdafxx7bovTcn/nz89WbvXsv4z3vWchuu73JbrstYLfd3mTIkEUbxfJ1G+Ofw85mH5bPPiyffdg5KtWP48ePn5ZSql9bvSKD9n7AJSmlI0vbXwJIKX19NfX7AY+llIZExKnAuJTSJ0v7/gNoSimtdupIfX19uu+++zr7Y/QITU1Nhcwtnj8/T8945JHWx8MPr3g7+P798wobbed/Dx+eV95YXQBftiyPqrYdoX7gAXj66RXP2zLdo2WUesSIYr6qb1FUP3YnS5fC3//eemHln/6Uvwnp1SsvJdhyYeUBB7Q/73l1fbhwIdx/f+uc6nvvzaPXkP+c7LFH6yj13nvnbyQ21hVT/HNYPvuwfPZh+ezDzlGpfoyIDgXtIqeO3AvsEhHDgOeAU4APt60QEdsAr6WUlgFfIq9AAvAb4GsR0fLl/hGl/epCttwS9tsvP9p6/fUVp5888ki+gca117bWGTBgxfnfVVWtofqhh1qnA/TqledNjx2b1wZuCdW1tU4JqISqqjzPub4+LyX49tt5hZuWaSZXXgn/9m95fvJ++7UG7733bg3G77yTf8Ztp4DMmNE61aeuLgfqs8/Oz2PHFvsLlCRJRSksaKeUlkTEOeTQXAVcm1J6JCIuBe5LKd0KjAO+HhGJPHXk7NKxr0XEV8hhHeDSlgsj1fUNHNi6vFxbr7yy4sj3I4/ADTfAlCl5/1Zb5RD9iU+0jlTvsUfnrwihzrPZZnDQQflxySX5F6S77mq9sPKSS+Dii/ONcvbfH+bMGcszz+SADrDNNjmEn3hi68WK221XyU8kSVLnKXR9hZTSbcBtK5Vd1Ob1DcANqzn2WlpHuNUDbLttXj6u7Tc8KeVVL5YtW/N0EnUPffvCkUfmB+RvN5qacuj+4x/z3OpzzmmdBlJX589cktRz9fCFzNTVRcDgwZVuhYoycCAcf3x+ADQ1TXdOoiRpo7ERXKMvSZIkbXgGbUmSJKkABm1JkiSpAAZtSZIkqQAGbUmSJKkABm1JkiSpAAZtSZIkqQAGbUmSJKkABm1JkiSpAAZtSZIkqQAGbUmSJKkABm1JkiSpAAZtSZIkqQAGbUmSJKkABm1JkiSpAAZtSZIkqQAGbUmSJKkABm1JkiSpAAZtSZIkqQAGbUmSJKkABm1JkiSpAAZtSZIkqQAGbUmSJKkABm1VVGMj1NVBr175ubGx0i2SJEnqHJtUugHaeDU2wsSJ0Nyct2fNytsADQ2Va5ckSVJncERbFTNpUmvIbtHcnMslSZK6O4O2Kmb27HUrlyRJ6k4M2qqYmpp1K5ckSepODNqqmMmTobp6xbLq6lwuSZLU3Rm0VTENDTBlCtTWQkR+njLFCyElSVLP4KojqqiGBoO1JEnqmRzRliRJkgpg0JYkSZIKYNCWJEmSCmDQliRJkgrQoaAd2T9FxEWl7ZqI2LvYpkmSJEndV0dHtL8L7AecWtpeAFxTSIskSZKkHqCjQXuflNLZwGKAlNLrwKZrOygiJkTE4xHxVESc387+moi4MyL+HhEPRsTRpfK6iFgUEdNLj++vw2eSJEmSKq6j62i/GxFVQAKIiG2BZWs6oFT/GuBwYC5wb0TcmlKa0abahcD1KaXvRcQewG1AXWnf0yml0R3+JJIkSVIX0tER7auAm4HtImIycBfwtbUcszfwVErpmZTSO8BU4NiV6iRgi9LrLYHnO9geSZIkqUuLlFLHKkbsBhwKBPD7lNKja6l/EjAhpXRGafs08hSUc9rUGQz8FhgI9AUOSylNi4g64BHgCeBN4MKU0p/aeY+JwESAQYMG7Tl16tQOfZaNzcKFC+nXr1+lm9Ht2Y/lsw/LZx+Wzz4sn31YPvuwc1SqH8ePHz8tpVS/tnodmjoSETsDz6aUromIccDhEfFCSumNNR3WTtnKqf5U4McppSsiYj/gZxExAngBqEkpvRoRewK3RMTwlNKbK5wspSnAFID6+vo0bty4jnycjU5TUxP2Tfnsx/LZh+WzD8tnH5bPPiyffdg5uno/dnTqyI3A0oh4D/CfwDDg52s5Zi4wtM32EFadGvIJ4HqAlNLdQB9gm5TS2ymlV0vl04Cngfd2sK2SJElSxXU0aC9LKS0BTgC+k1I6Fxi8lmPuBXaJiGERsSlwCnDrSnVmk6ejEBG7k4P2KxGxbeliSiJiJ2AX4JkOtlWSJEmquHVZdeRU4CPA+0tlvdd0QEppSUScA/wGqAKuTSk9EhGXAvellG4F/h/wg4g4lzyt5PSUUoqIg4BLI2IJsBT4VErptXX+dJIkSVKFdDRofwz4FDA5pfRsRAwD/mttB6WUbiMv2de27KI2r2cA+7dz3I3k6SpSxTU2wqRJMHv2wdTUwOTJ0NBQ6VZJkqSurkNBuxSIP9Nm+1ngsqIaJXUVjY0wcSI0NwMEs2blbTBsS5KkNevQHO2IOKZ098bXIuLNiFgQEW+u/Uipe5s0qSVkt2puzuWSJElr0tGpI98mXwj5UOrowttSDzB79rqVS5IktejoqiNzgIcN2drY1NSsW7kkSVKLjo5ofxG4LSL+ALzdUphSurKQVkldxOTJbedoZ9XVuVySJGlNOjqiPRloJq9z3b/NQ+rRGhpgyhSorYWIRG1t3vZCSEmStDYdHdHeKqV0RKEtkbqohob8aGr6Q5e+zaskSepaOjqifXtEGLQlSZKkDlpr0I6IIM/R/nVELHJ5P0mSJGnt1jp1pHRL9OkppbEbokGSJElST9DRqSN3R8RehbZEkiRJ6kE6ejHkeOBTETETeAsI8mD3qKIaJkmSJHVnHR3RPgrYCTgEeD9wTOlZUoU1NkJdHfTqlZ8bGyvdIkmSBB0c0U4pzSq6IZLWXWPjijfUmTUrb4NrfUuSVGkdHdGW1AVNmrTiXSshb0+aVJn2SJKkVgZtqRubPXvdyiVJ0oZj0Ja6sZqadSuXJEkbjkFb6sYmT4bq6hXLqqtzuSRJqiyDttSNNTTAlClQWwsR+XnKFC+ElCSpK+joOtqSuqiGBoO1JEldkSPakiRJUgEM2pIkSVIBDNqSJElSAQzakiRJUgEM2pIkSVIBDNqSCtfYCHV1cMghB1NXl7clSerpXN5PUqEaG2HiRGhuBghmzcrb4LKEkqSezRFtSYWaNKklZLdqbs7lkiT1ZAZtSYWaPXvdyiVJ6ikM2pIKVVOzbuWSJPUUBm1JhZo8GaqrVyyrrs7lkiT1ZAZtSYVqaIApU6C2FiIStbV52wshJUk9nUFbUuEaGmDmTLjjjj8wc6YhW5K0cTBoS5IkSQUwaEuSJEkFMGhLkiRJBTBoS5IkSQUoNGhHxISIeDwinoqI89vZXxMRd0bE3yPiwYg4us2+L5WOezwijiyynZI2bo2NUFcHvXrl58bGSrdIktQTbFLUiSOiCrgGOByYC9wbEbemlGa0qXYhcH1K6XsRsQdwG1BXen0KMBzYAbg9It6bUlpaVHslbZwaG2HixNbbxM+albfB1VEkSeUpckR7b+CplNIzKaV3gKnAsSvVScAWpddbAs+XXh8LTE0pvZ1SehZ4qnQ+SepUkya1huwWzc25XJKkckRKqZgTR5wETEgpnVHaPg3YJ6V0Tps6g4HfAgOBvsBhKaVpEXE18NeU0n+V6v0Q+FVK6YaV3mMiMBFg0KBBe06dOrWQz9LdLVy4kH79+lW6Gd2e/Vi+rtiHhxxyMCnFKuURiTvu+EMFWrRmXbEPuxv7sHz2Yfnsw85RqX4cP378tJRS/drqFTZ1BFj1f648gt3WqcCPU0pXRMR+wM8iYkQHjyWlNAWYAlBfX5/GjRtXXot7qKamJuyb8tmP5euKfVhTk6eLrFoeXa6t0DX7sLuxD8tnH5bPPuwcXb0fi5w6MhcY2mZ7CK1TQ1p8ArgeIKV0N9AH2KaDx0pS2SZPhurqFcuqq3O5JEnlKDJo3wvsEhHDImJT8sWNt65UZzZwKEBE7E4O2q+U6p0SEZtFxDBgF+CeAtsqaSPV0ABTpkBtLUTk5ylTvBBSklS+wqaOpJSWRMQ5wG+AKuDalNIjEXEpcF9K6Vbg/wE/iIhzyVNDTk950vgjEXE9MANYApztiiOSitLQYLCWJHW+Iudok1K6jbxkX9uyi9q8ngHsv5pjJwN+eStJkqRuyTtDSpIkSQUwaEuSJEkFMGhLkiRJBTBoS1I30NgIdXX5Bjt1dXlbktS1FXoxpCSpfI2NMHFiy63ig1mz8ja4WookdWWOaEtSFzdpUkvIbtXcnMslSV2XQVuSurjZs9etXJLUNRi0JamLq6lZt3JJUtdg0JakLm7yZKiuXrGsujqXS5K6LoO2JHVxDQ0wZQrU1kJEorY2b3shpCR1bQZtSeoGGhpg5ky4444/MHOmIVuSugODtiRJklQAg7YkSZJUAIO2JEmSVACDtiRJklQAg7YkSZJUAIO2JEmSVACDtiRJklQAg7YkqVM0NkJdHfTqlZ8bGyvdIkmqrE0q3QBJUvfX2AgTJ0Jzc96eNStvgzfXkbTxckRbklS2SZNaQ3aL5uZcLkkbK4O2JKlss2evW7kkbQwM2pKkstXUrFu5JG0MDNqSpLJNngzV1SuWVVfncknaWBm0JUlla2iAKVOgthYi8vOUKV4IKWnj5qojkqRO0dBgsJakthzRliRJkgpg0JYkSZIKYNCWJEmSCmDQliRJkgpg0JYkSZIKYNCWJEmSCmDQliRJkgpg0JYkbRQaG6GuDg455GDq6vK2JBXJG9ZIknq8xkaYOBGamwGCWbPyNniTHUnFcURbktTjTZrUErJbNTfnckkqikFbktTjzZ69buWS1BkKDdoRMSEiHo+IpyLi/Hb2fysippceT0TEG232LW2z79Yi2ylJ6tlqatatXJI6Q2FztCOiCrgGOByYC9wbEbemlGa01Ekpndum/qeBMW1OsSilNLqo9kmSNh6TJ7edo51VV+dySSpKkSPaewNPpZSeSSm9A0wFjl1D/VOB6wpsjyRpI9XQAFOmQG0tRCRqa/O2F0JKKlKklIo5ccRJwISU0hml7dOAfVJK57RTtxb4KzAkpbS0VLYEmA4sAS5LKd3SznETgYkAgwYN2nPq1KmFfJbubuHChfTr16/Szej27Mfy2Yflsw/LZx+Wzz4sn33YOSrVj+PHj5+WUqpfW70il/eLdspWl+pPAW5oCdklNSml5yNiJ+COiHgopfT0CidLaQowBaC+vj6NGzeuE5rd8zQ1NWHflM9+LJ99WD77sHz2Yfnsw/LZh52jq/djkVNH5gJD22wPAZ5fTd1TWGnaSErp+dLzM0ATK87fliRJkrq0IoP2vcAuETEsIjYlh+lVVg+JiF2BgcDdbcoGRsRmpdfbAPsDM1Y+VpIkSeqqCps6klJaEhHnAL8BqoBrU0qPRMSlwH0ppZbQfSowNa04WXx34D8iYhn5l4HL2q5WIkmSJHV1hd6CPaV0G3DbSmUXrbR9STvH/QUYWWTbJEmSpCJ5Z0hJkiSpAAZtSZIkqQAGbUmSuojGRqirg1698nNjY6VbJKkchc7RliRJHdPYuOJt4mfNytvgHSyl7soRbUmSuoBJk1pDdovm5lwuqXsyaEuS1AXMnr1u5ZK6PoO2JEldQE3NupVL6voM2pIkdQGTJ0N19Ypl1dW5XFL3ZNCWJKkLaGiAKVOgthYi8vOUKV4IKXVnrjoiSVIX0dBgsJZ6Eke0JUmSpAIYtCVJkqQCGLQlSZKkAhi0JUmSpAIYtCVJkqQCGLQlSZKkAhi0JUmSpAIYtCVJUoc0NkJdHRxyyMHU1eVtSavnDWskSdJaNTbCxInQ3AwQzJqVt8Gb7Eir44i2JElaq0mTWkJ2q+bmXC6pfQZtSZK0VrNnr1u5JIO2JEnqgJqadSuXZNCWJEkdMHkyVFevWFZdncsltc+gLUmS1qqhAaZMgdpaiEjU1uZtL4SUVs+gLUmSOqShAWbOhDvu+AMzZxqypbUxaEuSJEkFMGhLkiRJBTBoS5IkSQUwaEuSJEkFMGhLkiRJBTBoS5IkSQUwaEuSJEkFMGhLkqQeo7ER6uqgV6/83NhY6RZpY7ZJpRsgSZLUGRobYeJEaG7O27Nm5W3w5jqqDEe0JUlSjzBpUmvIbtHcnMulSjBoS5KkHmH27HUrl4pm0JYkST1CTc26lUtFKzRoR8SEiHg8Ip6KiPPb2f+tiJheejwREW+02ffRiHiy9Phoke2UJEnd3+TJUF29Yll1dS6XKqGwiyEjogq4BjgcmAvcGxG3ppRmtNRJKZ3bpv6ngTGl11sBFwP1QAKmlY59vaj2SpKk7q3lgsdJk/J0kZqaHLK9EFKVUuSI9t7AUymlZ1JK7wBTgWPXUP9U4LrS6yOB36WUXiuF698BEwpsqyRJ6gEaGmDmTFi2LD8bslVJkVIq5sQRJwETUkpnlLZPA/ZJKZ3TTt1a4K/AkJTS0oj4PNAnpfTV0v4vA4tSSpevdNxEYCLAoEGD9pw6dWohn6W7W7hwIf369at0M7o9+7F89mH57MPy2Yflsw/LZx92jkr14/jx46ellOrXVq/IdbSjnbLVpfpTgBtSSkvX5diU0hRgCkB9fX0aN27cejSz52tqasK+KZ/9WD77sHz2Yfnsw/LZh+WzDztHV+/HIqeOzAWGttkeAjy/mrqn0DptZF2PlSRJkrqcIoP2vcAuETEsIjYlh+lbV64UEbsCA4G72xT/BjgiIgZGxEDgiFKZJEmS1C0UNnUkpbQkIs4hB+Qq4NqU0iMRcSlwX0qpJXSfCkxNbSaLp5Rei4ivkMM6wKUppdeKaqskSZLU2Yqco01K6TbgtpXKLlpp+5LVHHstcG1hjZMkSZIK5J0hJUmSpAIYtCVJkqQCGLQlSZI2kMZGqKuDQw45mLq6vK2eq9A52pIkScoaG2HiRGhuBghmzcrb4B0seypHtCVJkjaASZNaQnar5uZcrp7JoC1JkrQBzJ69buXq/gzakiRJG0BNzbqVq/szaEuSJG0AkydDdfWKZdXVuVw9k0FbkiRpA2hogClToLYWIhK1tXnbCyF7LoO2JEnSBtLQADNnwh13/IGZMw3ZPZ1BW5IkSSqAQVuSJEkqgEFbkiRJKoBBW5IkSSqAQVuSJEkqgEFbkiRJKoBBW5IkSSqAQVuSJEnLNTZCXR306pWfGxsr3aLua5NKN0CSJEldQ2MjTJwIzc15e9asvA3eXGd9OKItSZIkACZNag3ZLZqbc7nWnUFbkiRJAMyevW7lWjODtiTp/7d3/8Fy1eUdx9+fEhUCRWAotBqSQAe0FKiBCAqKBAuTKgO0MkUmUlqstLSASrHKMOM4dmixMNXO1JFGpDhDSoZSfiqFUCBgBQEJv0NRSoBGaIMVKT86QMjTP/aglyv3JnFz7tmTvF8zd+7ud8+e87lPcvc++93v7pEkAGbOXL9xTc5GW5IkSQCceSZMn/7asenTB+NafzbakiRJAgZveFy4EGbNgmTwfeFC3wj58/JTRyRJkvQTCxbYWG8ozmhLkiRJLbDRliRJklpgoy1JkiS1wEZbkiRJaoGNtiRJktQCG21JkiSpBTbakiRJ6pVFi2D2bDj44Pcxe/bg+ijyc7QlSZLUG4sWwQknwAsvAITHHhtch9H7/G9ntCVJktQbZ5zxapP9Uy+8MBgfNTbakiRJ6o3HH1+/8S7ZaEuSJKk3Zs5cv/Eu2WhLkiSpN848E6ZPf+3Y9OmD8VHTaqOdZH6Sh5I8nOQzE2zzu0mWJ3kgyT+OGX8lyd3N15Vt5pQkSVI/LFgACxfCrFmQFLNmDa6P2hshocVPHUmyGfBl4BBgJXBHkiuravmYbXYFTgcOqKqnk+wwZhf/V1XvaCufJEmS+mnBgsHX0qU3cdBBB3UdZ0JtzmjvCzxcVY9U1UvAYuCIcdt8DPhyVT0NUFWrWswjSZIkTZlUVTs7To4C5lfVHzbXjwX2q6qTxmxzOfA94ABgM+BzVXVNc9tq4G5gNXBWVV3+Osc4ATgBYMcdd9xn8eLFrfwsfffcc8+x1VZbdR2j96zj8Kzh8Kzh8Kzh8Kzh8KzhhtFVHefNm3dnVc1d23ZtnrAmrzM2vqufBs2RyfsAAAlUSURBVOwKHATMAL6VZI+q+jEws6qeSLILcEOS+6rqP16zs6qFwEKAuXPn1ii/dNClpUuXjvTLKn1hHYdnDYdnDYdnDYdnDYdnDTeMUa9jm0tHVgI7jbk+A3jidba5oqperqoVwEMMGm+q6onm+yPAUmBOi1klSZKkDarNRvsOYNckOyd5I/BhYPynh1wOzANIsj2wG/BIkm2TvGnM+AHAciRJkqSeaG3pSFWtTnIScC2D9dfnV9UDST4PfLeqrmxuOzTJcuAV4FNV9T9J9gf+PskaBk8Gzhr7aSWSJEnSqGtzjTZVdTVw9bixz465XMCpzdfYbW4B9mwzmyRJktQmzwwpSZIktcBGW5IkSWqBjbYkSZLUAhttSZIkqQU22pIkSVILWjsF+1RL8hTwWNc5RtT2wA+7DrERsI7Ds4bDs4bDs4bDs4bDs4YbRld1nFVVv7S2jTaaRlsTS/LdqprbdY6+s47Ds4bDs4bDs4bDs4bDs4YbxqjX0aUjkiRJUgtstCVJkqQW2GhvGhZ2HWAjYR2HZw2HZw2HZw2HZw2HZw03jJGuo2u0JUmSpBY4oy1JkiS1wEZbkiRJaoGN9kYsyU5JbkzyYJIHkny860x9lWSzJHcl+UbXWfooyTZJLkny783/x3d3nalvknyy+T2+P8lFSTbvOlMfJDk/yaok948Z2y7JdUm+33zftsuMo26CGp7d/D7fm+SyJNt0mXHUvV4Nx9x2WpJKsn0X2fpiohomOTnJQ83j4193lW8iNtobt9XAn1XVrwHvAv40ye4dZ+qrjwMPdh2ix/4WuKaq3g78BtZyvSR5K3AKMLeq9gA2Az7cbareuACYP27sM8D1VbUrcH1zXRO7gJ+t4XXAHlW1F/A94PSpDtUzF/CzNSTJTsAhwONTHaiHLmBcDZPMA44A9qqqXwfO6SDXpGy0N2JV9WRVLWsuP8uguXlrt6n6J8kM4IPAeV1n6aMkWwMHAl8DqKqXqurH3abqpWnAFkmmAdOBJzrO0wtVdTPwo3HDRwBfby5/HThySkP1zOvVsKqWVNXq5up3gBlTHqxHJvh/CPBF4M8BP5liLSao4YnAWVX1YrPNqikPthY22puIJLOBOcBt3SbppS8xeCBc03WQntoFeAr4h2b5zXlJtuw6VJ9U1Q8YzNQ8DjwJPFNVS7pN1Ws7VtWTMJiQAHboOE/fHQ/8S9ch+ibJ4cAPquqerrP02G7Ae5PcluSmJO/sOtB4NtqbgCRbAf8MfKKq/rfrPH2S5DBgVVXd2XWWHpsG7A18parmAM/jS/XrpVlDfASwM/AWYMskH+k2lQRJzmCwTHFR11n6JMl04Azgs11n6blpwLYMlsd+Crg4SbqN9Fo22hu5JG9g0GQvqqpLu87TQwcAhyd5FFgMHJzkwm4j9c5KYGVVvfpqyiUMGm+tu98EVlTVU1X1MnApsH/Hmfrsv5P8CkDzfeRebu6DJMcBhwELypNyrK9fZfDE+Z7m78sMYFmSX+40Vf+sBC6tgdsZvPI8Um8qtdHeiDXP6r4GPFhVf9N1nj6qqtOrakZVzWbw5rMbqsqZxPVQVf8F/GeStzVD7weWdxipjx4H3pVkevN7/X58Q+kwrgSOay4fB1zRYZZeSjIf+DRweFW90HWevqmq+6pqh6qa3fx9WQns3Txeat1dDhwMkGQ34I3ADztNNI6N9sbtAOBYBrOwdzdfH+g6lDZJJwOLktwLvAP4y47z9ErzasAlwDLgPgaP3SN92uFRkeQi4FbgbUlWJvkocBZwSJLvM/jEh7O6zDjqJqjh3wG/CFzX/G05t9OQI26CGmo9TFDD84Fdmo/8WwwcN2qvrngKdkmSJKkFzmhLkiRJLbDRliRJklpgoy1JkiS1wEZbkiRJaoGNtiRJktQCG21J6oEkS5PMnYLjnJLkwSRTeqa/JJ9LctpUHlOS2jat6wCSpHYlmVZVq9dx8z8BfquqVrSZSZI2Bc5oS9IGkmR2Mxv81SQPJFmSZIvmtp/MSCfZvjntMkl+P8nlSa5KsiLJSUlOTXJXku8k2W7MIT6S5JYk9yfZt7n/lknOT3JHc58jxuz3n5JcBSx5naynNvu5P8knmrFzgV2AK5N8ctz2myU5uznOvUn+qBk/KMnNSS5LsjzJuUl+obntmCT3Ncf4wph9zU+yLMk9Sa4fc5jdmzo9kuSUMT/fN5tt709y9DD/RpI0lZzRlqQNa1fgmKr6WJKLgQ8BF67lPnsAc4DNgYeBT1fVnCRfBH4P+FKz3ZZVtX+SAxmcEW0P4Azghqo6Psk2wO1J/rXZ/t3AXlX1o7EHS7IP8AfAfkCA25LcVFV/3Jxae15VjT+N8UeBZ6rqnUneBHw7yasN/L7A7sBjwDXA7yS5BfgCsA/wNLAkyZHAt4GvAgdW1YpxTyTeDsxjcMbBh5J8BZgPPFFVH2yyv3kttZSkkWGjLUkb1oqquru5fCcwex3uc2NVPQs8m+QZ4Kpm/D5grzHbXQRQVTcn2bpprA8FDh+zvnlzYGZz+brxTXbjPcBlVfU8QJJLgfcCd02S8VBgryRHNdffzOBJxUvA7VX1SLOvi5r9vwwsraqnmvFFwIHAK8DNry5NGZfvm1X1IvBiklXAjk0NzmlmxL9RVd+aJKMkjRQbbUnasF4cc/kVYIvm8mp+ulxv80nus2bM9TW89nG6xt2vGMxIf6iqHhp7Q5L9gOcnyJiJwk8iwMlVde244xw0Sa6J9jN++1eNr920qvpeMwP/AeCvkiypqs+vb3hJ6oJrtCVpajzKYBkFwFGTbDeZowGSvIfBMo5ngGuBk5OkuW3OOuznZuDIJNOTbAn8NrC2meJrgROTvKE5zm7NfQH2TbJzszb7aODfgNuA9zXr0TcDjgFuAm5txndu9rPd+AONleQtwAtVdSFwDrD3Ovx8kjQSnNGWpKlxDnBxkmOBG37OfTzdrH3eGji+GfsLBmu4722a7UeBwybbSVUtS3IBcHszdF5VTbZsBOA8BstgljXHeQo4srntVuAsYE8GTfxlVbUmyenAjQxmsa+uqisAkpwAXNo05quAQyY57p7A2UnWMFiOcuJackrSyEjVRK/gSZI0uWbpyGlVNWlzL0mbIpeOSJIkSS1wRluSJElqgTPakiRJUgtstCVJkqQW2GhLkiRJLbDRliRJklpgoy1JkiS14P8B9we7huxB3ygAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x504 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_learning_curve(history, 'rmse')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MLP model testing\n",
    "And finally, make a prediction and check the testing error using out-of-sample data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The out-of-sample RMSE of rating predictions is 0.9094\n"
     ]
    }
   ],
   "source": [
    "# load best model\n",
    "MLP_model = get_MLP_model(num_users, num_items, [64, 32, 16, 8], [0, 0, 0, 0])\n",
    "MLP_model = load_trained_model(MLP_model, os.path.join(data_path, 'tmp/model.hdf5'))\n",
    "# make prediction using test data\n",
    "predictions = MLP_model.predict([df_test.userId.values, df_test.movieId.values])\n",
    "# get the RMSE\n",
    "error = rmse(df_test.rating.values, predictions)\n",
    "print('The out-of-sample RMSE of rating predictions is', round(error, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Train Neural Matrix Factorization (NeuMF) and Test Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### define NeuMF model architeture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_NeuMF_model(num_users, num_items, MF_dim, MF_reg, MLP_layers, MLP_regs):\n",
    "    \"\"\"\n",
    "    Build Neural Matrix Factorization (NeuMF) Model Topology.\n",
    "    This is stack version of both GMF and MLP\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    num_users: int, total number of users\n",
    "    num_iterms: int, total number of items\n",
    "    MF_dim: int, embedded dimension for user vector and item vector in MF\n",
    "    MF_reg: tuple of float, L2 regularization of MF embedded layer\n",
    "    MLP_layers: list of int, each element is the number of hidden units for each MLP layer,\n",
    "        with the exception of first element. First element is the sum of dims of\n",
    "        user latent vector and item latent vector\n",
    "    MLP_regs: list of int, each element is the L2 regularization parameter for\n",
    "        each layer in MLP\n",
    "\n",
    "    Return\n",
    "    ------\n",
    "    A Keras Model with MLP model architeture\n",
    "    \"\"\"\n",
    "    assert len(MLP_layers) == len(MLP_regs)\n",
    "    num_MLP_layer = len(MLP_layers) # Number of layers in the MLP\n",
    "    # Input variables\n",
    "    user_input = Input(shape=(1,), dtype='int32', name='user_input')\n",
    "    item_input = Input(shape=(1,), dtype='int32', name='item_input')\n",
    "\n",
    "    # Embedding layer\n",
    "    \n",
    "    # MF\n",
    "    MF_Embedding_User = Embedding(\n",
    "        input_dim=num_users + 1,\n",
    "        output_dim=MF_dim,\n",
    "        embeddings_initializer='uniform',\n",
    "        name='mf_user_embedding',\n",
    "        embeddings_regularizer=l2(MF_reg[0]),\n",
    "        input_length=1)\n",
    "    MF_Embedding_Item = Embedding(\n",
    "        input_dim=num_items + 1,\n",
    "        output_dim=MF_dim,\n",
    "        embeddings_initializer='uniform',\n",
    "        name='mf_item_embedding',\n",
    "        embeddings_regularizer=l2(MF_reg[1]),\n",
    "        input_length=1)\n",
    "    \n",
    "    # MLP\n",
    "    MLP_Embedding_User = Embedding(\n",
    "        input_dim=num_users + 1,\n",
    "        output_dim=MLP_layers[0] // 2,\n",
    "        embeddings_initializer='uniform',\n",
    "        name='mlp_user_embedding',\n",
    "        embeddings_regularizer=l2(MLP_regs[0]),\n",
    "        input_length=1)\n",
    "    MLP_Embedding_Item = Embedding(\n",
    "        input_dim=num_items + 1,\n",
    "        output_dim=MLP_layers[0] // 2,\n",
    "        embeddings_initializer='uniform',\n",
    "        name='mlp_item_embedding',\n",
    "        embeddings_regularizer=l2(MLP_regs[0]),\n",
    "        input_length=1) \n",
    "    \n",
    "    # MF part\n",
    "    mf_user_latent = Flatten()(MF_Embedding_User(user_input))\n",
    "    mf_item_latent = Flatten()(MF_Embedding_Item(item_input))\n",
    "    mf_vector = Multiply()([mf_user_latent, mf_item_latent])\n",
    "\n",
    "    # MLP part\n",
    "    mlp_user_latent = Flatten()(MLP_Embedding_User(user_input))\n",
    "    mlp_item_latent = Flatten()(MLP_Embedding_Item(item_input))\n",
    "    mlp_vector = Concatenate(axis=-1)([mlp_user_latent, mlp_item_latent])\n",
    "    for idx in range(1, num_MLP_layer):\n",
    "        layer = Dense(\n",
    "            units=MLP_layers[idx],\n",
    "            activation='relu',\n",
    "            kernel_initializer='glorot_uniform',\n",
    "            kernel_regularizer=l2(MLP_regs[idx]),\n",
    "            name = 'layer%d' %idx)\n",
    "        mlp_vector = layer(mlp_vector)\n",
    "    \n",
    "    # Concatenate MF and MLP parts\n",
    "    predict_vector = Concatenate(axis=-1)([mf_vector, mlp_vector])\n",
    "\n",
    "    # Final prediction layer\n",
    "    prediction = Dense(1, kernel_initializer='glorot_uniform', name='prediction')(predict_vector)\n",
    "    \n",
    "    # Stitch input and output\n",
    "    model = Model([user_input, item_input], prediction)\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### create NeuMF model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "user_input (InputLayer)         (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "item_input (InputLayer)         (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "mlp_user_embedding (Embedding)  (None, 1, 32)        21504       user_input[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "mlp_item_embedding (Embedding)  (None, 1, 32)        290144      item_input[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "flatten_18 (Flatten)            (None, 32)           0           mlp_user_embedding[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "flatten_19 (Flatten)            (None, 32)           0           mlp_item_embedding[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_6 (Concatenate)     (None, 64)           0           flatten_18[0][0]                 \n",
      "                                                                 flatten_19[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "mf_user_embedding (Embedding)   (None, 1, 10)        6720        user_input[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "mf_item_embedding (Embedding)   (None, 1, 10)        90670       item_input[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "layer1 (Dense)                  (None, 32)           2080        concatenate_6[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "flatten_16 (Flatten)            (None, 10)           0           mf_user_embedding[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "flatten_17 (Flatten)            (None, 10)           0           mf_item_embedding[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "layer2 (Dense)                  (None, 16)           528         layer1[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "multiply_4 (Multiply)           (None, 10)           0           flatten_16[0][0]                 \n",
      "                                                                 flatten_17[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "layer3 (Dense)                  (None, 8)            136         layer2[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_7 (Concatenate)     (None, 18)           0           multiply_4[0][0]                 \n",
      "                                                                 layer3[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "prediction (Dense)              (None, 1)            19          concatenate_7[0][0]              \n",
      "==================================================================================================\n",
      "Total params: 411,801\n",
      "Trainable params: 411,801\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "NeuMF_model = get_NeuMF_model(\n",
    "    num_users=num_users,\n",
    "    num_items=num_items,\n",
    "    MF_dim=10,\n",
    "    MF_reg=(0, 0),\n",
    "    MLP_layers=[64, 32, 16, 8],\n",
    "    MLP_regs=[0, 0, 0, 0])\n",
    "NeuMF_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### train NeuMF model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Kevin/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:108: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60002 samples, validate on 20001 samples\n",
      "Epoch 1/30\n",
      "60002/60002 [==============================] - 8s 127us/step - loss: 1.3704 - mean_squared_error: 1.3704 - rmse: 1.0858 - val_loss: 0.8790 - val_mean_squared_error: 0.8790 - val_rmse: 0.9326\n",
      "Epoch 2/30\n",
      "60002/60002 [==============================] - 7s 109us/step - loss: 0.8107 - mean_squared_error: 0.8107 - rmse: 0.8957 - val_loss: 0.8304 - val_mean_squared_error: 0.8304 - val_rmse: 0.9059\n",
      "Epoch 3/30\n",
      "60002/60002 [==============================] - 7s 109us/step - loss: 0.7424 - mean_squared_error: 0.7424 - rmse: 0.8568 - val_loss: 0.8137 - val_mean_squared_error: 0.8137 - val_rmse: 0.8968\n",
      "Epoch 4/30\n",
      "60002/60002 [==============================] - 6s 104us/step - loss: 0.7066 - mean_squared_error: 0.7066 - rmse: 0.8353 - val_loss: 0.8190 - val_mean_squared_error: 0.8190 - val_rmse: 0.9001\n",
      "Epoch 5/30\n",
      "60002/60002 [==============================] - 6s 107us/step - loss: 0.6818 - mean_squared_error: 0.6818 - rmse: 0.8203 - val_loss: 0.8347 - val_mean_squared_error: 0.8347 - val_rmse: 0.9079\n",
      "Epoch 6/30\n",
      "60002/60002 [==============================] - 7s 120us/step - loss: 0.6593 - mean_squared_error: 0.6593 - rmse: 0.8075 - val_loss: 0.8271 - val_mean_squared_error: 0.8271 - val_rmse: 0.9041\n",
      "Epoch 7/30\n",
      "60002/60002 [==============================] - 6s 106us/step - loss: 0.6367 - mean_squared_error: 0.6367 - rmse: 0.7930 - val_loss: 0.8488 - val_mean_squared_error: 0.8488 - val_rmse: 0.9162\n",
      "Epoch 8/30\n",
      "60002/60002 [==============================] - 6s 106us/step - loss: 0.6154 - mean_squared_error: 0.6154 - rmse: 0.7798 - val_loss: 0.8483 - val_mean_squared_error: 0.8483 - val_rmse: 0.9153\n",
      "Epoch 9/30\n",
      "60002/60002 [==============================] - 6s 103us/step - loss: 0.5925 - mean_squared_error: 0.5925 - rmse: 0.7649 - val_loss: 0.8531 - val_mean_squared_error: 0.8531 - val_rmse: 0.9180\n",
      "Epoch 10/30\n",
      "60002/60002 [==============================] - 6s 102us/step - loss: 0.5658 - mean_squared_error: 0.5658 - rmse: 0.7474 - val_loss: 0.8789 - val_mean_squared_error: 0.8789 - val_rmse: 0.9324\n",
      "Epoch 11/30\n",
      "60002/60002 [==============================] - 7s 111us/step - loss: 0.5388 - mean_squared_error: 0.5388 - rmse: 0.7289 - val_loss: 0.9005 - val_mean_squared_error: 0.9005 - val_rmse: 0.9433\n",
      "Epoch 12/30\n",
      "60002/60002 [==============================] - 7s 120us/step - loss: 0.5117 - mean_squared_error: 0.5117 - rmse: 0.7102 - val_loss: 0.8981 - val_mean_squared_error: 0.8981 - val_rmse: 0.9420\n",
      "Epoch 13/30\n",
      "60002/60002 [==============================] - 7s 115us/step - loss: 0.4815 - mean_squared_error: 0.4815 - rmse: 0.6889 - val_loss: 0.9209 - val_mean_squared_error: 0.9209 - val_rmse: 0.9540\n",
      "Epoch 00013: early stopping\n"
     ]
    }
   ],
   "source": [
    "# model config\n",
    "BATCH_SIZE = 64\n",
    "EPOCHS = 30\n",
    "VAL_SPLIT = 0.25\n",
    "\n",
    "# train model\n",
    "history = train_model(NeuMF_model, 'adam', BATCH_SIZE, EPOCHS, VAL_SPLIT, \n",
    "                      inputs=[df_train.userId.values, df_train.movieId.values],\n",
    "                      outputs=df_train.rating.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Matrix Factorization (NeuMF) learning curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtoAAAG5CAYAAACwZpNaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3XuclWW9///XhxHE4SSCkoLD4KmUQ4Cj6NYUzANaqakZNFmWStttu3Z2sii1vlHutpa1y2wqv22/e5LtDw9Z2bbtxsksVMTQEhVPgHhGRRlHlMP1++New6wZBhhgbtaamdfz8ViPte7rvu57XWtdD4b3XHPd1x0pJSRJkiR1rl6lboAkSZLUHRm0JUmSpBwYtCVJkqQcGLQlSZKkHBi0JUmSpBwYtCVJkqQcGLQlqRNFRHVEpIjYqQN1z46Iu3ZEu7ZHRDRGxD6lbockdTUGbUk9VkQsiYi3I2Jom/KFhbBcXZqWbV1gz1tKqX9K6ck8zh0RB0TE/xcRKyLitYh4MCIujIiKPN5PknYkg7aknu4pYHrzRkSMBXYpXXN2rFIG2ojYF7gHeBoYm1IaBHwIqAEGbMP5Sv5LiSQVM2hL6un+H/Cxou2PA9cWV4iIQRFxbUS8FBFLI+JrEdGrsK8iIi4vjMg+CbyvnWN/ERHPRcQzEfGt7Q23EdErIi6KiCci4uWIuD4idiva//9FxPOFEeI7I2J00b5fRsRPIuLWiHgDmFIo+3FE/C4iVkXEPYUQ3HxMioj9io7fXN3jI+LRwntfFRF/jIhzN/FRvgH8JaV0YUrpOYCU0qMppY+klFZGxOSIWN7msy+JiGMLry+NiDkR8Z8R8Trw1Yh4s813MaHQN70L25+MiIcj4tWIuC0iRm57T0jS5hm0JfV0dwMDI+LAQgD+MPCfber8OzAI2Ac4miyYf6Kw7zzg/cAEspHYM9oc+x/AWmC/Qp3jgU0Fz476DHBqoS17Aa8CPy7a/3tgf2AP4H6gvs3xHwFmkY0aN88Rn04WfAcDjxf2b0q7dQtTcOYAXwGGAI8C/7CZ8xxbqL89TimcY1fg34B5wOlF+z8CzEkprYmIU4GvAqcBuwN/Aq7bzveXpE0yaEtSy6j2ccAjwDPNO4rC91dSSqtSSkuAK4CzClXOBK5MKT2dUnoF+E7RscOAE4F/SSm9kVJ6Efg+MG072/spYGZKaXlK6S3gUuCM5qkTKaVrCm1t3vfuiBhUdPyvU0p/TimtTymtLpTdmFK6N6W0liyYj9/M+2+q7knAQymlGwv7fgg8v5nzDAGe25oP3o55KaWbC5/lTeBXFKYCRUSQfde/KtT9FPCdlNLDhfZ9GxjvqLakvDifTZKyoH0nMIo200aAoUAfYGlR2VJgeOH1XmRzjIv3NRsJ9AaeyzIfkA1wFNffFiOBmyJifVHZOmBYRDxPNsL8IbJR2+Y6Q4HXCq/be//iQNwE9N/M+2+qbqvvIqWU2k79aONlYM/N7O+Itp9lDvDvEbEX2ah+Ihu5hux7+0FEXFFUP8j6cimS1Mkc0ZbU46WUlpJdFHkScGOb3SuANWQhrVkVLaPezwF7t9nX7GngLWBoSmnXwmNgSmk02+dp4MSic+6aUuqbUnqGbKrEKWTTMgYB1YVjouj4tJ3vvynPASOaNwojyiM2XZ3baT3No603gMqi81WQ/fJQrNVnSSmtBP5A9peGjwDXpZSa6zwNfKrN97ZLSukvm/9YkrRtDNqSlDkHOCal9EZxYUppHXA9MCsiBhSmGVxIyzzu64HPRMSIiBgMXFR07HNkoe+KiBhYuIhx34g4eivatXNE9C169AKuLrRnJEBE7B4RpxTqDyAL9y+ThdRvb93XsF1+B4yNiFML01guAN6xmfqXAP8QEf8WEe8AiIj9Chc37gosBvpGxPsKFzN+Ddi5A+34FdlUoNNpmTYC2ff2leaLQwsXqn5oKz+jJHWYQVuSgJTSEyml+zax+5/JRlefJLt48FfANYV9PwNuAx4gu/Cw7Yj4x8imniwiu2hxDls3XaIReLPocQzwA+AW4A8RsYrsgs5JhfrXkk2DeKbwnndvxXttl5TSCrIpK98lC/oHAfeRBf/26j8BHE426v5QRLwG3FA4ZlVK6TXgn4Cfk32eN4DNTUVpdgvZtJEXUkoPFL3fTcC/ArMLq5T8nWwOvSTlIlr+oiZJUucpjL4vB2pTSneUuj2StKM5oi1J6jQRcUJE7BoRO5MtpRfswFF1SSonBm1JUmc6HHiC7CLSDwCnFpbdk6QeJ7epIxFxDdlNHF5MKY1pZ3+QzTM8iWx5qLNTSvcX9q0D/laouiyldHIujZQkSZJykueI9i+BqZvZfyLZxSr7AzOAnxTtezOlNL7wMGRLkiSpy8nthjUppTsjonozVU4Bri2sb3p3YU7fnoXlsLba0KFDU3X15t5OHfHGG2/Qr1+/UjdDReyT8mS/lB/7pPzYJ+XJftl+CxYsWJFSaruu/0ZKeWfI4bS+o9fyQtlzZOum3gesBS5LKd3c3gkiYgbZaDjDhg3j8ssvz7fFPUBjYyP9+2/uhnDa0eyT8mS/lB/7pPzYJ+XJftl+U6ZM6dDdZEsZtKOdsuYJ41UppWcjYh9gbkT8rbDeauvKKdUBdQA1NTVp8uTJuTW2p2hoaMDvsbzYJ+XJfik/9kn5sU/Kk/2y45Ry1ZHltL5t8QjgWYCUUvPzk0ADMGFHN06SJEnaHqUM2rcAH4vMYcBrKaXnImJwYf1VImIocATZ3c0kSZKkLiO3qSMRcR0wGRgaEcuBS4DeACmlq4FbyZb2e5xseb9PFA49EPhpRKwn+0XgspSSQVuSJHUpa9asYfny5axevbrUTWll0KBBPPzww6VuRpfQt29fRowYQe/evbfp+DxXHZm+hf0JuKCd8r8AY/NqlyRJ0o6wfPlyBgwYQHV1NdntQ8rDqlWrGDBgQKmbUfZSSrz88sssX76cUaNGbdM5vDOkJElSDlavXs2QIUPKKmSr4yKCIUOGbNdfJAzakiRJOTFkd23b238GbUmSJCkHBm1JkqRuaOXKlVx11VXbdOxJJ53EypUrN1vn4osv5vbbb9+m8/cUBm1JkqQyUF8P1dXQq1f2XF+/fefbXNBet27dZo+99dZb2XXXXTdb55vf/CbHHnvsNrdvU7bUtq7EoC1JklRi9fUwYwYsXQopZc8zZmxf2L7ooot44oknGD9+PF/84hdpaGhgypQpfPKTn2Ts2GyBt1NPPZWDDz6Y0aNHU1dXt+HY6upqVqxYwZIlSzjwwAM577zzGD16NMcffzxvvvkmAGeffTZz5szZUP+SSy5h4sSJjB07lkceeQSAl156ieOOO46JEyfyqU99ipEjR7JixYqN2tq/f38uvvhiJk2axLx586iuruarX/0qhx9+ODU1Ndx///2ccMIJ7Lvvvlx99dUAPPfccxx11FGMHz+eMWPG8Kc//QmAP/zhDxx++OFMnDiRD33oQzQ2Nm77l7idDNqSJEklNnMmNDW1Lmtqysq31WWXXca+++7LwoUL+bd/+zcA7r33Xi6++GIWLcpuUXLNNdewYMEC7rvvPn74wx/y8ssvb3Sexx57jAsuuICHHnqIXXfdlRtuuKHd9xs6dCj3338/559/PpdffjkA3/jGNzjmmGO4//77+eAHP8iyZcvaPfaNN95gzJgx3HPPPRx55JEA7L333sybN4/3vOc9G0L93XffzcUXXwzAr371K0444QQWLlzIAw88wPjx41mxYgXf+ta3uP3227n//vupqanhe9/73rZ/idspt3W0JUmS1DGbyJ+bLN9Whx56KNXV1Ru2f/jDH3LTTTcB8PTTT/PYY48xZMiQVseMGjWK8ePHA3DwwQezZMmSds992mmnbahz4403AnDXXXdtOP/UqVMZPHhwu8dWVFRw+umntyo7+eSTARg7diyNjY0MGDCAAQMG0LdvX1auXMkhhxzCJz/5SdasWcOpp57K+PHj+eMf/8iiRYs44ogjAHj77bc5/PDDO/r1dDpHtLdDZ8+lkiRJPVNV1daVb6t+/fpteN3Q0MDtt9/OvHnzeOCBB5gwYUK7a0bvvPPOG15XVFSwdu3ads/dXK+4TnZ/wi3r27cvFRUV7Z6vV69erdrQq1cv1q5dy1FHHcWdd97J8OHDOeuss7j22mtJKXHcccexcOFCFi5cyKJFi/jFL37RoTbkwaC9jfKYSyVJknqmWbOgsrJ1WWVlVr6tBgwYwKpVqza5/7XXXmPw4MFUVlbyyCOPcPfdd2/7m23CkUceyfXXXw9kc6dfffXVTjv30qVL2WOPPTjvvPM455xzuP/++znssMP485//zOOPPw5AU1MTixcv7rT33FoG7W2Ux1wqSZLUM9XWQl0djBwJEdlzXV1Wvq2GDBnCEUccwZgxY/jiF7+40f6pU6eydu1axo0bx9e//nUOO+yw7fgE7bvkkkv4wx/+wMSJE/n973/Pnnvu2Wm3f29oaGD8+PFMmDCBG264gc9+9rPsvvvu/PKXv2T69OmMGzeOww47bMOFmaUQHR3SL3c1NTXpvvvu22Hv16tXNpLdVgSsX7/DmtHpGhoamDx5cqmboSL2SXmyX8qPfVJ+enqfPPzwwxx44IGlbsZGVq1a1Wlhd0veeustKioq2GmnnZg3bx7nn38+Cxcu3CHv3Vna68eIWJBSqtnSsV4MuY2qqrLpIu2VS5IkCZYtW8aZZ57J+vXr6dOnDz/72c9K3aQdyqC9jWbNyuZkF08f2d65VJIkSd3J/vvvz1//+tdSN6NknKO9jfKYSyVJkqTuwxHt7VBba7CWJElS+xzRliRJknJg0JYkSZJyYNCWJEkSAP379wfg2Wef5Ywzzmi3zuTJk9nSkspXXnklTUUrRpx00kmsXLmy8xraRRi0JUmS1Mpee+3FnDlztvn4tkH71ltvZdddd+2Mpm2wqVvBlxODtiRJUjf05S9/mauuumrD9qWXXsoVV1xBY2Mj733ve5k4cSJjx47l17/+9UbHLlmyhDFjxgDw5ptvMm3aNMaNG8eHP/xh3nzzzQ31zj//fGpqahg9ejSXXHIJAD/84Q959tlnmTJlClOmTAGgurqaFStWAPC9732PMWPGMGbMGK688soN73fggQdy3nnnMXr0aI4//vhW79Ps7LPP5sILL2TKlCl8+ctf5tJLL+XjH/84xx9/PNXV1dx444186UtfYuzYsUydOpU1a9YAcNFFF3HQQQcxbtw4vvCFLwDw0ksvcfrpp3PIIYdwyCGH8Oc//3m7v/O2XHVEkiQpZ//yL9DZN0QcPx4KObVd06ZN41/+5V/4p3/6JwCuv/56/vu//5u+ffty0003MXDgQFasWMFhhx3GySefTES0e56f/OQnVFZW8uCDD/Lggw8yceLEDftmzZrFbrvtxrp163jve9/Lgw8+yGc+8xm+973vcccddzB06NBW51qwYAH/9//+X+655x5SSkyaNImjjz6awYMH89hjj3Hdddfxs5/9jDPPPJMbbriBj370oxu1Z/Hixdx+++1UVFRw6aWX8sQTT3DHHXewaNEiDj/8cG644Qa++93v8sEPfpDf/e53HHXUUdx000088sgjRMSGKSyf/exn+dznPseRRx7JsmXLOOGEE3j44Ye3ths2yxFtSZKkbmjChAm8+OKLPPvsszzwwAMMHjyYqqoqUkp89atfZdy4cRx77LE888wzvPDCC5s8z5133rkh8I4bN45x48Zt2Hf99dczceJEJkyYwEMPPcSiRYs226a77rqLD37wg/Tr14/+/ftz2mmn8ac//QmAUaNGMX78eAAOPvhglixZ0u45PvShD1FRUbFh+8QTT6R3796MHTuWdevWMXXqVADGjh3LkiVLGDhwIH379uXcc8/lxhtvpLKyEoDbb7+dT3/604wfP56TTz6Z119/nVWrVm3hW906jmhLkiTlbHMjz3k644wzmDNnDs8//zzTpk0DsnD80ksvsWDBAnr37k11dTWrV6/e7HnaG+1+6qmnuPzyy5k/fz6DBw/m7LPP3uJ5Ukqb3LfzzjtveF1RUdHu1BGAfv36tXtcr1696N2794a29urVi7Vr17LTTjtx77338r//+7/Mnj2bH/3oR8ydO5f169czb948dtlll822eXs4oi1JktRNTZs2jdmzZzNnzpwNq4i89tpr7LHHHvTu3Zs77riDpUuXbvYcRx11FPX19QD8/e9/58EHHwTg9ddfp1+/fgwaNIgXXniB3//+9xuOGTBgQLujw0cddRQ333wzTU1NvPHGG9x000285z3v6ayP267GxkZee+01TjrpJK688koWFubwHH/88fzoRz/aUG9hZ8/twRFtSZKkbmv06NGsWrWK4cOHs+eeewLw4Q9/mOnTp1NTU8P48eN517vetdlznH/++XziE59g3LhxjB8/nkMPPRSAd7/73UyYMIHRo0ezzz77cMQRR2w4ZsaMGZx44onsueee3HHHHRvKJ06cyNlnn73hHOeeey4TJkzY5DSRzrBq1SpOOeUUVq9eTUqJ73//+0B20eYFF1zAuHHjWLt2LUcddRRXX311p753bG4IvyupqalJW1rTUVvW0NDA5MmTS90MFbFPypP9Un7sk/LT0/vk4Ycf5sADDyx1MzayatUqBgwYUOpmdBnt9WNELEgp1WzpWKeOSJIkSTkwaEuSJEk5MGhLkiTlpLtM0e2ptrf/DNqSJEk56Nu3Ly+//LJhu4tKKfHyyy/Tt2/fbT6Hq45IkiTlYMSIESxfvpyXXnqp1E1pZfXq1dsVHnuSvn37MmLEiG0+3qAtSZKUg969ezNq1KhSN2MjDQ0NTJgwodTN6BGcOiJJkiTlwKAtSZIk5cCgLUmSJOXAoC1JkiTlILegHRHXRMSLEfH3TeyPiPhhRDweEQ9GxMSifR+PiMcKj4/n1UZJkiQpL3mOaP8SmLqZ/ScC+xceM4CfAETEbsAlwCTgUOCSiBicYzslSZKkTpdb0E4p3Qm8spkqpwDXpszdwK4RsSdwAvA/KaVXUkqvAv/D5gO7JEmSVHZKuY72cODpou3lhbJNlW8kImaQjYYzbNgwGhoacmloT9LY2Oj3WGbsk/Jkv5Qf+6T82CflyX7ZcUoZtKOdsrSZ8o0LU6oD6gBqamrS5MmTO61xPVVDQwN+j+XFPilP9kv5sU/Kj31SnuyXHaeUq44sB/Yu2h4BPLuZckmSJKnLKGXQvgX4WGH1kcOA11JKzwG3AcdHxODCRZDHF8okSZKkLiO3qSMRcR0wGRgaEcvJVhLpDZBSuhq4FTgJeBxoAj5R2PdKRPwfYH7hVN9MKW3uokpJkiSp7OQWtFNK07ewPwEXbGLfNcA1ebRLkiRJ2hG8M6QkSZKUA4O2JEmSlAODtiRJkpQDg7YkSZKUA4O2JEmSlAODtiRJkpQDg7YkSZKUA4O2JEmSlAODtiRJkpQDg7YkSZKUA4O2JEmSlAODtiRJkpQDg7YkSZKUA4O2JEmSlAODtiRJkpQDg7YkSZKUA4O2JEmSlAODtiRJkpQDg7YkSZKUA4O2JEmSlAODtiRJkpQDg7YkSZKUA4O2JEmSlIOdSt0ASZIkqaNWr4ann4Zly+Cgg2DPPUvdok0zaEuSJKkspASvvJKF6KVL239+4YWW+v/xH/Cxj5WuvVti0JYkSdIOsXYtPPNMS2huL0i/8UbrY3bZBaqqYORIePe7s+eqquwxdmxpPkdHGbQlSZLUKVat2vxo9DPPwPr1rY/ZffcsPB94IJxwQkuQbn4eOhQiSvN5tpdBW5IkSVu0fn02baNteC5+/eqrrY/ZaSfYe+8sNE+ZsnGI3ntvqKwszefZEQzakiRJ2nCR4aZGo59+Gt5+u/Uxgwa1hOYjjtg4SL/jHVBRUZrPUw4M2pIkST3EY4/BXXcN5YEHNg7TL77Yum4E7LVXFpoPOQTOOKN1iK6qyoK2Ns2gLUmS1E2tXw/33gu//jXcfDM88gjAGCC7yLA5NI8f3xKem8tGjIDevUva/C7PoC1JktSNvPUWzJ2bhetf/xqefz6bK3300XDBBdC7932cfnoNQ4Z03YsMuwqDtiRJUhe3ciXcems2av3730NjI/TvDyeeCKecAiedBIMHZ3UbGhoZOrS07e0pDNqSJEld0NNPwy23ZOG6oSFbo3rYMPjIR7Jwfcwx0LdvqVvZsxm0JUmSuoCU4KGHsmB9882wYEFW/s53wuc/n4XrSZOgV6/StlMtDNqSJEllat06+MtfWsL1k09m5YcdBpddloXrd72rtG3Uphm0JUmSykhTE9x+exasf/MbWLEC+vSB974Xvvxl+MAHYM89S91KdYRBW5IkqcRWrIDf/jZbJeS22+DNN7M1qt/3Pjj1VJg6FQYMKHUrtbVyDdoRMRX4AVAB/DyldFmb/SOBa4DdgVeAj6aUlhf2rQP+Vqi6LKV0cp5tlSRJ2pGeeqplfes//Slb83rECDjnnGxKyNFHu451V5db0I6ICuDHwHHAcmB+RNySUlpUVO1y4NqU0n9ExDHAd4CzCvveTCmNz6t9kiRJO1JK8Ne/toTrBx/MyseOhZkzs3A9caJrW3cneY5oHwo8nlJ6EiAiZgOnAMVB+yDgc4XXdwA359geSZKkHWrNGrjzzixY//rX2ZJ8vXrBkUfC976Xhet99il1K5WXSCnlc+KIM4CpKaVzC9tnAZNSSp8uqvMr4J6U0g8i4jTgBmBoSunliFgLLATWApellDYK4RExA5gBMGzYsINnz56dy2fpSRobG+nfv3+pm6Ei9kl5sl/Kj31SfnpqnzQ1VTB//m7cdddQ7r57Nxobe7PzzuuoqXmVI49cweGHv8ygQWtK1r6e2i+dacqUKQtSSjVbqpfniHZ7f/hom+q/APwoIs4G7gSeIQvWAFUppWcjYh9gbkT8LaX0RKuTpVQH1AHU1NSkyZMnd2Lze6aGhgb8HsuLfVKe7JfyY5+Un57UJ88/n60QcvPN8L//m90GfcgQOOOM7GLG446roLJyKFD6WzL2pH4ptTyD9nJg76LtEcCzxRVSSs8CpwFERH/g9JTSa0X7SCk9GRENwASgVdCWJEkqlcWLW9a3vvvubA72qFHwT/+Uhet/+AfYyfXderQ8u38+sH9EjCIbqZ4GfKS4QkQMBV5JKa0HvkK2AgkRMRhoSim9VahzBPDdHNsqSZK0WevXw/z5LeH6kUey8oMPhm98IwvXY8Z4MaNa5Ba0U0prI+LTwG1ky/tdk1J6KCK+CdyXUroFmAx8JyIS2dSRCwqHHwj8NCLWA73I5mgv2uhNJEmScrR8eTZaffvtcMst8Nxz2Sj15MlwwQVw8slQVVXqVqpc5foHjZTSrcCtbcouLno9B5jTznF/Acbm2TZJkqRiTU2wYEEWrO+5J3t+5plsX//+cOKJ2SohJ50EgweXtq3qGpw5tJ1S8k9EkiR1NSnBY4+1DtUPPADr1mX79903u2HMYYdlj3e/O7sNurQ1DNrbISWYNg1Gj4aLLvIfoCRJ5erVV+Hee1uC9T33wCuvZPsGDIBJk7L/yw87LHu9++6lba+6B4P2dnj77WzR+UsugTlz4Be/gEMOKXWrJEnq2dauhb//PQvVzcG6+cLFiOyCxdNOaxmtfte7oKKitG1W92TQ3g477wzXXQfTp8P552f/WC+8MLvyuLKy1K2TJKlneO65llB9991w333ZfGuAPfbI/n8+66zsuaYGBg4sbXvVcxi0O8HJJ2fzuL74Rbj88mzJn5//PCuTJEmdZ/VquP/+1qPVy5Zl+3r3hokT4bzzsukfhx0G1dVeS6XSMWh3kkGDoK4um7N93nnZsj//+I/wr//qb86SJG2LlODJJ1uPVj/wAKwp3L28ujq7KcznPpeF6vHjoW/fkjZZasWg3cmOOQb+9jf4+tfhyivht7+Fq6+G972v1C2TJKm8vfZadkOY4tHqFSuyff36waGHwhe+0HLB4rBhpW2vtCUG7RxUVsIVV8CZZ8I558D73w+1tVnwHjq01K2TJKn01q2DRYtaj1Y//HA2ig1w0EHZ1MzmUD16tBcsqusxaOdo0qRs4fvvfAdmzYLbboN//3f48IedLyZJ6lleeKFlveq7785Grhsbs31DhmSBevr07PmQQ7IpmVJXZ9DO2c47w6WXwumnwyc/mf0Que46uOoqGD681K2TJGn7pARvvJFN+1i5svXzXXeN4Kc/zYL1kiVZ/Z12yuZSn312y2j1vvs6AKXuyaC9g4wdC/PmwQ9+AF/7WvYnscsvh3PP9YeLJKl03n47C8XtBeXm5y3ta76b4sb2Y++9s0D9z/+cPU+YALvssiM/oVQ6Bu0daKed4POfh1NOyQL2jBnZ6PbPfpb9Ni9J0tZYvx5Wrdq+oPzmm1t+n0GDWh677pr9RXb06Jbt4n3Fz4888mdOPfWI/L8IqUwZtEtgv/1g7txsre0vfCEb7Z41Cz7zGS/0kKSeat06eOKJ7ALBl17qWFB+/fWWiwc3pW/fjQNwVdXGZZsKzAMGZHdB3hbPP79m2w6UugmDdon06pWNaJ90Urbe9oUXwn/9V3Yb99GjS906SVJeUsouDPzb31o/Fi3aeHS5V6+Ng++oUZsfRS7eN2hQdq2QpNIwaJfYiBHwm9/A7NnZiPaECdkc7osugj59St06SdL2aGyEv/+9JUw3v25eGxqytaDHjs0GXcaOhTFjYM89s8Dcr5/X8UhdmUG7DERkq5Ece2wWti+5BObMyUa3Dzmk1K2TJG3J2rWwePHGo9RPPdVSp1+/LESfckoWqJsfu+9eunZLypdBu4zsvnt2ceT06XD++dnV2RdeCN/4RnYTHElSaaUEy5dvPEL98MPZ6h2QXWtzwAHZQMknP9kSqKurt32us6SuyaBdhk4+GY4+Gr74xWwJwJtvzlYmmTy51C2TpJ5j5crW0z6ag/XKlS11hg/PQvTxx7cE6ne9K7sAUZIM2mVq0CCoq4Np0+C882DKFPjUp+C734WBA0vdOknqPt56Cx55ZONQ/fTTLXUGDsxC9Ic/3Hrax+DBpWu3pPJn0C5zxxyT/cD/+tfhyivhd7+Dq6+G972v1C2TpK5l/XpE0iayAAAgAElEQVRYunTjedSLF2dzrAF6985GpN/zntaBeu+9vShR0tYzaHcBlZVwxRVw5plwzjnw/vdDbW0WvIcOLXXrJKn8rFjR/rSPxsaWOiNHZiG6+OLEAw5wxSdJnceg3YVMmgT33w/f/nb2uO02+Pd/z/6U6UiLpJ5o/Xp4/HG491645ZZ9+fa3s1D9/PMtdXbbLQvRZ5/dsnzemDFOw5OUP4N2F9OnD1x6KZx+eja6PX16tlLJVVdlF+VIUnf2/PNZqG5+zJ/fcnFi797DGTOm9YWJY8dma1I7GCGpFAzaXdTYsTBvXjZ95Gtfg4MOylYoOfdc/0OR1D00NsJ997UO1s0XKFZUZD8HzzwTDj00e7z44p9473uPLm2jJamIQbsLq6iAz38+m1947rnZLd2vuy5bCnDffUvdOknquDVrsjnUxaF60aJsaghktx0/4oiWUD1hwsb3F2hoSDu+4ZK0GQbtbmC//WDuXPj5z+ELX8hGeb71LfjsZ7MwLknlJCV48snWofr++2H16mz/kCFZmD799Oz5kEO8e6Kkrsmg3U306pWNaJ90EvzjP2Yj3f/1X3DNNTB6dKlbJ6kne+mlbC71Pfe0BOtXXsn29e0LBx+c3Q23ebR61CinwEnqHgza3cyIEfCb38Ds2fCZz2R/Xv3a1+Cii1yySlL+mpqy0eni0eqnnsr2RWS/+H/wgy2hevTobO1qSeqODNrdUES2Gsmxx2bTRy65BObMgV/8IvsTrCR1hnXr4KGHWofqv/89KweoqsrCdPNo9cSJMGBAadssSTuSQbsb2313+NWvstB9/vlw2GFw4YXwjW9sfBGRpPatWwfLlsGjj2Z3EFy8OLu74KpVB7HffjBoEOy6a/a8uUdXH7VNKfseikP1ggXwxhvZ/l13zcL0Bz7QMq/6He8obZslqdQM2j3ABz4ARx0FX/pStgTgzTdnK5NMnlzqlknlISV48cWWIF38ePxxePvtlroDB0J1Naxc2Y/Fi+G117LpEltSWbnlML6lwL7TDvyJ/cor2bzq4mD94ovZvp13zqalnXNOyxSQ/fZzXrUktWXQ7iEGDYKf/hSmTcuWApwyBT71KfjXf832ST3BqlXw2GOtg3TzSPXrr7fU69MnC44HHADvf3/23PzYY48sUDY0zGdy4bfVNWuywL2lx8qVrV8vXdqy/eabW25/ZWXHRs83VWfgwPbD+urVsHBhS6C+557sFwzIPuu73gUnnpgF6kmTspWNvOZDkrbMoN3DTJmS3Z7461/Pbnbz299mAfx97yt1y6TO8fbb2cV3xSG6+fHccy31IrI5xAccAGed1RKk3/nOrHxrlsbs3RuGDs0e29Pu119vHca3FNpfeSX7rM1lzcvjbU6/fq0D+VtvwYMPwtq12f7hw7NA3TxaffDB/jIuSdvKoN0DVVbCFVdkd1Q755xsxO4jH4Ef/KDULZM6Zv16eOaZ9qd6PPVUy8V4kIXfAw6AE07IQnRzoN53X9hll9J9hrb69OmcsL65UfT2Hv37wxe/2DKvevjwzvtMktTTGbR7sEmTsmW4vv3t7HHbbTB69GgOPTSbg1r86NevtG1Vz/TKK+2H6cceaz0vurIyC88TJ2bTo5rD9P77w267la79O1qfPtlF0N7cRZLKg0G7h+vTBy69NLsD28UXw4IF/bjnnuzPycV2333j8D1qVPY8cqSrmGjbvflmNh+47ZzpxYvh5Zdb6lVUwD77ZAH6mGNaz5sePtwL8SRJ5cegLSC7uOmmm6Ch4V6OOmoyL7wAS5Zs/Fi4EH7969arMEB2gVhx+C5+jBxZXn+i1461dm02feHll7NpHW3nTS9b1rr+Xntl4fn001vPmx41qusvkSdJ6lkM2tpIr16w557Z4/DDN96/fj08/3zrAP7UU9nzggVw443ZKgzF3vGO9kfDq6uzC8/69s31I6kTrF/fcgHeyy9v/rn49cqVG59r0KAsQB91VOuR6f3284YmkqTuI9egHRFTgR8AFcDPU0qXtdk/ErgG2B14BfhoSml5Yd/Hga8Vqn4rpfQfebZVHderVzbquNde8A//sPH+9euz1R2KA3jzY/787C6VzSscNNtzz/ZHw5uD+M475/qRepSUstUtOhKSi59ffTU7dlMGD87mQ++2GwwZks2PHjKkZXu33bK/bhxwQDYVyakekqTuLregHREVwI+B44DlwPyIuCWltKio2uXAtSml/4iIY4DvAGdFxG7AJUANkIAFhWNfzau96jy9emVzZocPhyOO2Hj/unXw7LPtT02ZNw/+679arxoRkYX6TY2I7713z1zTNyVobOxYSG5bp/j7bWvgwNbheNSo1tvtPe+669YthydJUk+Q54j2ocDjKaUnASJiNnAKUBy0DwI+V3h9B3Bz4fUJwP+klF4pHPs/wFTguhzb26PV18PMmbBs2dFUVcGsWVBbm897VVRk4XjvveE979l4/9q1rYN48aj4n/8Ms2dvHMSHD28dvvfaKwv8KbWMwrb33BX2LV68D9de235objtFp1i/fq3D8Lhxmw7Lza8HD3YetCRJnSXPoD0ceLpoezkwqU2dB4DTyaaXfBAYEBFDNnHsRqu7RsQMYAbAsGHDaGho6Ky29yi3374Hl1/+Tt56qwIIli6Fc85Zx8MPP8qxx75Y0rZVVWWPo49uKVu3LnjppT48//wuPP9831aPP/yhLy+9tDPr13ePeQkRid69hzNw4FsMHLiGgQPXMHToWkaNWsOgQWsYMGAtAwasYdCg7Dmrk73u02cz8zyKrFkDL7yQPdRxjY2N/swpM/ZJ+bFPypP9suPkGbTbSzpt/+f/AvCjiDgbuBN4BljbwWNJKdUBdQA1NTWp+XbI2jpnn73xcn5vvVXBf/7nQXzrWweVpE3bY80aeOmllu2IlvnA7T2X476i1tPQ8KfCrb6dqF5OGhoa8GdOebFPyo99Up7slx0nz6C9HNi7aHsE8GxxhZTSs8BpABHRHzg9pfRaRCwHJrc5tiHHtvZobZdX21J5uevdO5s6IkmSVEq9cjz3fGD/iBgVEX2AacAtxRUiYmhENLfhK2QrkADcBhwfEYMjYjBwfKFMOaiq2rpySZIkbVluQTultBb4NFlAfhi4PqX0UER8MyJOLlSbDDwaEYuBYcCswrGvAP+HLKzPB77ZfGGkOt+sWRvf2bGyMiuXJEnStsl1He2U0q3ArW3KLi56PQeYs4ljr6FlhFs5al5dJFt1JFFVFbmuOiJJktQT5Dl1RF1IbW22fN7cuX9kyRJDtiRJ0vYyaEuSJEk5MGhLkiRJOTBoS5IkSTkwaEuSJEk5MGhLkiRJOTBoS5IkSTkwaEuSJEk5MGhLkiRJOTBoS5IkSTkwaEuSJEk5MGhLkiRJOTBoS5IkSTkwaEuSJEk5MGhLkiRJOTBoS5IkSTkwaEuSJEk5MGhLkiRJOTBoS5IkSTkwaEuSJEk5MGhLkiRJOTBoS5IkSTkwaEuSJEk5MGhLkiRJOTBoS5IkSTkwaEuSJEk5MGhLkiRJOTBoS5IkSTkwaEuSJEk5MGhLkiRJOTBoq1upr4fqaujVK3uury91iyRJUk+1U6kbIHWW+nqYMQOamrLtpUuzbYDa2tK1S5Ik9UyOaKvbmDmzJWQ3a2rKyiVJknY0g7a6jWXLtq5ckiQpTx0K2pH5aERcXNiuiohD822atHWqqrauXJIkKU8dHdG+CjgcmF7YXgX8OJcWSdto1iyorGxdVlmZlUuSJO1oHQ3ak1JKFwCrAVJKrwJ9cmuVtA1qa6GuDkaOhIjsua7OCyElSVJpdDRor4mICiABRMTuwPotHRQRUyPi0Yh4PCIuamd/VUTcERF/jYgHI+KkQnl1RLwZEQsLj6u34jOpB6uthSVLYP367NmQLUmSSqWjy/v9ELgJ2CMiZgFnAF/b3AGFYP5j4DhgOTA/Im5JKS0qqvY14PqU0k8i4iDgVqC6sO+JlNL4Dn8SSZIkqYx0KGinlOojYgHwXiCAU1NKD2/hsEOBx1NKTwJExGzgFKA4aCdgYOH1IODZrWi7JEmSVLYipbTlShH7AstTSm9FxGRgHHBtSmnlZo45A5iaUjq3sH0W2VzvTxfV2RP4AzAY6Accm1JaEBHVwEPAYuB14GsppT+18x4zgBkAw4YNO3j27Nkd+czajMbGRvr371/qZqiIfVKe7JfyY5+UH/ukPNkv22/KlCkLUko1W6rX0akjNwA1EbEf8HPgN8CvgJM2c0y0U9Y21U8HfplSuiIiDgf+X0SMAZ4DqlJKL0fEwcDNETE6pfR6q5OlVAfUAdTU1KTJkyd38ONoUxoaGvB7LC/2SXmyX8qPfVJ+7JPyZL/sOB29GHJ9SmktcBrwg5TS54A9t3DMcmDvou0RbDw15BzgeoCU0jygLzA0pfRWSunlQvkC4AnggA62VZIkSSq5rVl1ZDrwMeC3hbLeWzhmPrB/RIyKiD7ANOCWNnWWkc37JiIOJAvaL0XE7oWLKYmIfYD9gSc72FZJkiSp5DoatD9BdsOaWSmlpyJiFPCfmzugMAL+aeA24GGy1UUeiohvRsTJhWqfB86LiAeA64CzUzZp/CjgwUL5HOAfU0qvbO2HkyRJkkqlo6uOLAI+U7T9FHBZB467lWzJvuKyi9uc94h2jruBbF64JEmS1CV1aEQ7It5fuKnMKxHxekSsiojXt3ykJEmS1DN1dNWRK8kuhPxb6sh6gJIkSVIP19E52k8DfzdkS5IkSR3T0RHtLwG3RsQfgbeaC1NK38ulVZIkSVIX19GgPQtoJFt+r09+zZEkSZK6h44G7d1SSsfn2hJJkiSpG+noHO3bI8KgLe1A9fVQXQ3HHHM01dXZtiRJ6jq2OKIdEUE2R/tLEfEWsAYIIKWUBubcPqlHqq+HGTOgqQkgWLo02waorS1lyyRJUkdtcUS7sNLIwpRSr5TSLimlgSmlAYZsKT8zZzaH7BZNTVm5JEnqGjo6dWReRBySa0skbbBs2daVS5Kk8tPRoD0FuDsinoiIByPibxHxYJ4Nk3qyqqqtK5ckSeWno6uOnJhrKyS1MmtW8RztTGVlVi5JkrqGDgXtlNLSvBsiqUXzBY8zZ8KyZYmqqmDWLC+ElCSpK+no1BFJO1htLSxZAnPn/pElSwzZkiR1NQZtSZIkKQcGbUmSJCkHBm1JkiQpBwZtSZIkKQcGbUmSJCkHBm1JkiQpBwZtSZIkKQcGbUmSJCkHBm1Juauvh+pq6NUre66vL3WLJEnKX4duwS5J26q+HmbMgKambHvp0mwbvNulJKl7c0RbUq5mzmwJ2c2amrJySZK6M4O2pFwtW7Z15ZIkdRcGbUm5qqraunJJkroLg7akXM2aBZWVrcsqK7NySZK6M4O2pFzV1kJdHYwcCRHZc12dF0JKkro/Vx2RlLvaWoO1JKnncURbkiRJyoFBW5IkScqBQVuSJEnKgUFbkiRJyoFBW5IkScqBQVuSJEnKgUFbkiRJyoFBW5K2Qn09VFfDMcccTXV1ti1JUntyDdoRMTUiHo2IxyPionb2V0XEHRHx14h4MCJOKtr3lcJxj0bECXm2U5I6or4eZsyApUshpWDp0mzbsC1Jak9uQTsiKoAfAycCBwHTI+KgNtW+BlyfUpoATAOuKhx7UGF7NDAVuKpwPkkqmZkzoampdVlTU1YuSVJbeY5oHwo8nlJ6MqX0NjAbOKVNnQQMLLweBDxbeH0KMDul9FZK6Sng8cL5JKlkli3bunJJUs+2U47nHg48XbS9HJjUps6lwB8i4p+BfsCxRcfe3ebY4W3fICJmADMAhg0bRkNDQ2e0u0drbGz0eywz9kn52GOPw3jhhb7tlK+moeHudo7QjuS/lfJjn5Qn+2XHyTNoRztlqc32dOCXKaUrIuJw4P9FxJgOHktKqQ6oA6ipqUmTJ0/evhaLhoYG/B7Li31SPq64IpuTXTx9pLISrriir31UBvy3Un7sk/Jkv+w4eU4dWQ7sXbQ9gpapIc3OAa4HSCnNA/oCQzt4rCTtULW1UFcHI0dCRGLkyGy7trbULZMklaM8g/Z8YP+IGBURfcgubrylTZ1lwHsBIuJAsqD9UqHetIjYOSJGAfsD9+bYVknqkNpaWLIE5s79I0uWGLIlSZuW29SRlNLaiPg0cBtQAVyTUnooIr4J3JdSugX4PPCziPgc2dSQs1NKCXgoIq4HFgFrgQtSSuvyaqskSZLU2fKco01K6Vbg1jZlFxe9XgQcsYljZwGz8myfJEmSlBfvDClJkiTlwKAtSZIk5cCgLUmSJOXAoC1JkiTlwKAtSZIk5cCgLUmSJOXAoC1JPVB9PVRXQ69e2XN9falbJEndT67raEuSyk99PcyYAU1N2fbSpdk2eKdLSepMjmhLUg8zc2ZLyG7W1JSVS5I6j0FbknqYZcu2rlyStG0M2pLUw1RVbV25JGnbGLQlqYeZNQsqK1uXVVZm5ZKkzmPQlqQeprYW6upg5EiIyJ7r6rwQUpI6m6uOSFIPVFtrsJakvDmiLUmSJOXAoC1JkiTlwKAtSZIk5cCgLUmSJOXAoC1JkiTlwKAtSZIk5cCgLUmSJOXAoC1JkiTlwKAtSerS6uuhuhqOOeZoqquzbUkqB94ZUpLUZdXXw4wZ0NQEECxdmm2Dd76UVHqOaEuSuqyZM5tDdoumpqxckkrNoC1J6rKWLdu6cknakQzakqQuq6pq68olaUcyaEuSuqxZs6CysnVZZWVWLkmlZtCWJHVZtbVQVwcjR0JEYuTIbNsLISWVA4O2JKlLq62FJUtg7tw/smSJIVtS+TBoS5IkSTkwaEuSJEk5MGhLkiRJOTBoS5IkSTkwaEuSJEk5MGhLkiRJOTBoS5IkSTnINWhHxNSIeDQiHo+Ii9rZ//2IWFh4LI6IlUX71hXtuyXPdkqSJEmdbae8ThwRFcCPgeOA5cD8iLglpbSouU5K6XNF9f8ZmFB0ijdTSuPzap8kSZKUpzxHtA8FHk8pPZlSehuYDZyymfrTgetybI8kSZK0w+QZtIcDTxdtLy+UbSQiRgKjgLlFxX0j4r6IuDsiTs2vmZIkSVLny23qCBDtlKVN1J0GzEkprSsqq0opPRsR+wBzI+JvKaUnWr1BxAxgBsCwYcNoaGjohGb3bI2NjX6PZcY+KU/2S/npDn1y++178POf78OLL+7MHnu8xbnnPsmxx75Y6mZts+7QJ92R/bLj5Bm0lwN7F22PAJ7dRN1pwAXFBSmlZwvPT0ZEA9n87Sfa1KkD6gBqamrS5MmTO6PdPVpDQwN+j+XFPilP9kv56ep9Ul8P3/8+NDVl2y+80Jfvf/8gDjzwIGprS9u2bdXV+6S7sl92nDynjswH9o+IURHRhyxMb7R6SES8ExgMzCsqGxwROxdeDwWOABa1PVaSpO5i5syWkN2sqSkrl9Q15TainVJaGxGfBm4DKoBrUkoPRcQ3gftSSs2hezowO6VUPK3kQOCnEbGe7JeBy4pXK5EkqbtZtmzryiWVvzynjpBSuhW4tU3ZxW22L23nuL8AY/NsmyRJ5aSqCpYubb9cUtfknSElSSoDs2ZBZWXrssrKrFxS12TQliSpDNTWQl0djBwJEdlzXR1d9kJISTlPHZEkSR1XW2uwlroTR7QlSZKkHBi0JUmSpBwYtCVJkqQcGLQlSZKkHBi0JUmSpBwYtCVJkqQcGLQlSZKkHBi0JUmSpBwYtCVJkqQcGLQlSZKkHBi0JUmSpBwYtCVJUqeqr4fqajjmmKOprs62pZ5op1I3QJIkdR/19TBjBjQ1AQRLl2bbALW1pWyZtOM5oi1JkjrNzJnNIbtFU1NWLvU0Bm1JktRpli3bunKpOzNoS5KkTlNVtXXlUndm0JYkSZ1m1iyorGxdVlmZlUs9jUFbkiR1mtpaqKuDkSMhIjFyZLbthZDqiQzakiSpU9XWwpIlMHfuH1myxJCtnsugLUmSJOXAoC1JkiTlwKAtSZIk5cCgLUmSJOXAoC1JkiTlwKAtSZIk5cCgLUmSJOXAoC1JkiTlwKAtSZIk5cCgLUmSJOXAoC1JkiTlwKAtSZIk5cCgLUmSJOXAoC1JkiTlwKAtSZK0CfX1UF0NvXplz/X1pW6RupKdSt0ASZKkclRfDzNmQFNTtr10abYNUFtbunap68h1RDsipkbEoxHxeERc1M7+70fEwsJjcUSsLNr38Yh4rPD4eJ7tlCRJamvmzJaQ3aypKSuXOiK3Ee2IqAB+DBwHLAfmR8QtKaVFzXVSSp8rqv/PwITC692AS4AaIAELCse+mld7JUmSii1btnXlUlt5jmgfCjyeUnoypfQ2MBs4ZTP1pwPXFV6fAPxPSumVQrj+H2Bqjm2VJElqpapq68qltvKcoz0ceLpoezkwqb2KETESGAXM3cyxw9s5bgYwA2DYsGE0NDRsd6N7usbGRr/HMmOflCf7pfzYJ+Wnq/fJRz+6B5df/k7eeqtiQ9nOO6/jox99lIaGF0vYsu3T1fulK8kzaEc7ZWkTdacBc1JK67bm2JRSHVAHUFNTkyZPnrwNzVSxhoYG/B7Li31SnuyX8mOflJ+u3ieTJ8OBB2Zzspcty0ayZ82qoLb2IOCgUjdvm3X1fulK8gzay4G9i7ZHAM9uou404II2x05uc2xDJ7ZNkiRpi2prXWFE2y7POdrzgf0jYlRE9CEL07e0rRQR7wQGA/OKim8Djo+IwRExGDi+UCZJkiR1CbmNaKeU1kbEp8kCcgVwTUrpoYj4JnBfSqk5dE8HZqeUUtGxr0TE/yEL6wDfTCm9kldbJUmSpM6W6w1rUkq3Are2Kbu4zfalmzj2GuCa3BonSZIk5chbsEuSJEk5MGhLkiRJOTBoS5IkSTkwaEuSJEk5MGhLkiRJOTBoS5IkSTkwaEuSJEk5MGhLkiRJOTBoS5Ik9QD19VBdDcccczTV1dm28pXrnSElSZJUevX1MGMGNDUBBEuXZtsAtbWlbFn35oi2JElSNzdzZnPIbtHUlJUrPwZtSZKkbm7Zsq0rV+cwaEuSJHVzVVVbV67OYdCWJEnq5mbNgsrK1mWVlVm58mPQliRJ6uZqa6GuDkaOhIjEyJHZthdC5sugLUmS1APU1sKSJTB37h9ZssSQvSMYtCVJkqQcGLQlSZKkHBi0JUmSpBwYtCVJkqQcGLQlSZKkHBi0JUmSpBwYtCVJkqQcGLQlSZKkHBi0JUmSpBwYtCVJkqQcGLQlSZKkHBi0JUmSpBwYtCVJktSl1NdDdTX06pU919eXukXt26nUDZAkSZI6qr4eZsyApqZse+nSbBugtrZ07WqPI9qSJEnqMmbObAnZzZqasvJyY9CWJElSl7Fs2daVl5JBW5IkSV1GVdXWlZeSQVuSJEldxqxZUFnZuqyyMisvNwZtSZIkdRm1tVBXByNHQkT2XFdXfhdCgquOSJIkqYuprS3PYN2WI9qSJElSDgzakiRJUg5yDdoRMTUiHo2IxyPiok3UOTMiFkXEQxHxq6LydRGxsPC4Jc92SpIkSZ0ttznaEVEB/Bg4DlgOzI+IW1JKi4rq7A98BTgipfRqROxRdIo3U0rj82qfJEmSlKc8R7QPBR5PKT2ZUnobmA2c0qbOecCPU0qvAqSUXsyxPZIkSdIOEymlfE4ccQYwNaV0bmH7LGBSSunTRXVuBhYDRwAVwKUppf8u7FsLLATWApellG5u5z1mADMAhg37/9u7/1ir6zqO489XXBOF/DXLpWjghppDCiV/R5A/RuqU0k1dmqXrh+XvLHNurdlWOlnZ1qYpkW4ymBn4O4FExPytaICQWoBFWGAq+WNTkVd/nA96OMKFku/5Hu95Pba7+z3f8/3xOue9e+77fs7nnu9O+02ZMqWSx9JNXn31VQYOHFh3jGiSmnSm1KXzpCadJzXpTKnL+zdmzJjHbY/c2HZVfryf1rOutavvAYYCo4FBwH2Shtl+GdjN9nJJuwOzJM23/dd1DmZfA1wDMHLkSI8ePXozP4TuM3v2bPI8dpbUpDOlLp0nNek8qUlnSl3ap8qpI8uAXZtuDwKWr2ebW2y/ZXsJ8DSNxhvby8v3xcBsYESFWSMiIiIiNqsqG+1HgaGShkj6MHAS0PrpITcDYwAk7QjsASyWtL2kLZvWHwIsJCIiIiLiA6KyqSO2V0s6C5hOY/71RNtPSboUeMz2reW+IyUtBN4Gvmf735IOBn4laQ2NPwYua/60koiIiIiITlfpJdht3wnc2bLuh03LBi4oX83bPADsU2W2iIiIiIgq5cqQEREREREVqOzj/dpN0krgubpz9AE7Ai/UHSLWkZp0ptSl86QmnSc16Uypy/v3Cdsf3dhGfabRjs1D0mOb8rmQ0T6pSWdKXTpPatJ5UpPOlLq0T6aORERERERUII12REREREQF0mhHq2vqDhDvkZp0ptSl86QmnSc16UypS5tkjnZERERERAUyoh0RERERUYE02hERERERFUijHUjaVdI9khZJekrSuXVnigZJ/SQ9Ien2urNEg6TtJN0k6c/lZ+agujN1O0nnl9euBZImS+pfd6ZuJGmipBWSFjSt20HSTEnPlu/b15mxG22gLleU17B5kqZJ2q7OjH1ZGu0AWA181/YngQOB70jau+ZM0XAusKjuELGOXwB32d4L+BSpT60k7QKcA4y0PQzoB5xUb6qudR0wtmXdD4C7bQ8F7i63o72u4711mQkMsz0ceAa4uN2hukUa7cD287bnluVXaDQOu9SbKiQNAo4GJtSdJRokbQOMAn4NYPtN2y/XmyqAHmArST3A1sDymvN0JdtzgBdbVh8HXF+WrwfGtTVUrLcutmfYXl1uPgQManuwLpFGO9YhaTAwAni43iQBXAl8H1hTd5B4x+7ASuA3ZUrPBEkD6g7VzWz/AxgP/A14Hlhle0a9qaLJTrafh8agDvCxmvPEe50O/L7uEH1VGu14h7KvjocAAAUrSURBVKSBwO+A82z/p+483UzSMcAK24/XnSXW0QPsC1xlewTwGnkrvFZlzu9xwBBgZ2CApFPqTRXxwSDpEhrTRyfVnaWvSqMdAEjagkaTPcn21LrzBIcAx0paCkwBPi/phnojBbAMWGZ77Ts+N9FovKM+hwNLbK+0/RYwFTi45kzxrn9J+jhA+b6i5jxRSDoNOAb4snNRlcqk0Q4kicac00W2f1Z3ngDbF9seZHswjX/smmU7o3Q1s/1P4O+S9iyrDgMW1hgpGlNGDpS0dXktO4z8g2onuRU4rSyfBtxSY5YoJI0FLgKOtf163Xn6sjTaAY3R01NpjJo+Wb6OqjtURIc6G5gkaR7waeAnNefpauXdhZuAucB8Gr/XcnnpGkiaDDwI7ClpmaQzgMuAIyQ9CxxRbkcbbaAuvwQ+Aswsv/OvrjVkH5ZLsEdEREREVCAj2hERERERFUijHRERERFRgTTaEREREREVSKMdEREREVGBNNoRERERERVIox0R8QEgabakkW04zzmSFklq65XiJP1I0oXtPGdERNV66g4QERHVktRje/Umbv5t4Au2l1SZKSKiG2REOyJiM5E0uIwGXyvpKUkzJG1V7ntnRFrSjpKWluWvSrpZ0m2Slkg6S9IFkp6Q9JCkHZpOcYqkByQtkLR/2X+ApImSHi37HNd03N9Kug2YsZ6sF5TjLJB0Xll3NbA7cKuk81u27yfpinKeeZK+WdaPljRH0jRJCyVdLelD5b6TJc0v57i86VhjJc2V9CdJdzedZu/yPC2WdE7T47ujbLtA0onvp0YREe2UEe2IiM1rKHCy7a9LuhE4HrhhI/sMA0YA/YG/ABfZHiHp58BXgCvLdgNsHyxpFDCx7HcJMMv26ZK2Ax6R9Iey/UHAcNsvNp9M0n7A14ADAAEPS7rX9rfKpZnH2H6hJeMZwCrbn5G0JXC/pLUN/P7A3sBzwF3AlyQ9AFwO7Ae8BMyQNA64H7gWGGV7ScsfEnsBY2hcse5pSVcBY4Hlto8u2bfdyHMZEdEx0mhHRGxeS2w/WZYfBwZvwj732H4FeEXSKuC2sn4+MLxpu8kAtudI2qY01kcCxzbNb+4P7FaWZ7Y22cWhwDTbrwFImgp8Fniil4xHAsMlnVBub0vjj4o3gUdsLy7HmlyO/xYw2/bKsn4SMAp4G5izdmpKS747bL8BvCFpBbBTeQ7GlxHx223f10vGiIiOkkY7ImLzeqNp+W1gq7K8mnen6/XvZZ81TbfXsO7rtFv2M40R6eNtP918h6QDgNc2kFEbCt8LAWfbnt5yntG95NrQcVq3X6v1ueux/UwZgT8K+KmkGbYv/V/DR0TUIXO0IyLaYymNaRQAJ/SyXW9OBJB0KI1pHKuA6cDZklTuG7EJx5kDjJO0taQBwBeBjY0UTwfOlLRFOc8eZV+A/SUNKXOzTwT+CDwMfK7MR+8HnAzcCzxY1g8px9mh9UTNJO0MvG77BmA8sO8mPL6IiI6QEe2IiPYYD9wo6VRg1v95jJfK3OdtgNPLuh/TmMM9rzTbS4FjejuI7bmSrgMeKasm2O5t2gjABBrTYOaW86wExpX7HgQuA/ah0cRPs71G0sXAPTRGse+0fQuApG8AU0tjvgI4opfz7gNcIWkNjekoZ24kZ0REx5C9oXfwIiIielemjlxou9fmPiKiG2XqSEREREREBTKiHRERERFRgYxoR0RERERUII12REREREQF0mhHRERERFQgjXZERERERAXSaEdEREREVOC/ENep4n5+QuEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x504 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_learning_curve(history, 'rmse')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Matrix Factorization (NeuMF) model testing\n",
    "And finally, make a prediction and check the testing error using out-of-sample data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The out-of-sample RMSE of rating predictions is 0.9059\n"
     ]
    }
   ],
   "source": [
    "# load best model\n",
    "NeuMF_model = get_NeuMF_model(\n",
    "    num_users=num_users,\n",
    "    num_items=num_items,\n",
    "    MF_dim=10,\n",
    "    MF_reg=(0, 0),\n",
    "    MLP_layers=[64, 32, 16, 8],\n",
    "    MLP_regs=[0, 0, 0, 0])\n",
    "NeuMF_model = load_trained_model(NeuMF_model, os.path.join(data_path, 'tmp/model.hdf5'))\n",
    "# make prediction using test data\n",
    "predictions = NeuMF_model.predict([df_test.userId.values, df_test.movieId.values])\n",
    "# get the RMSE\n",
    "error = rmse(df_test.rating.values, predictions)\n",
    "print('The out-of-sample RMSE of rating predictions is', round(error, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Observations\n",
    "1. ALS's out-of-sample RMSE is 0.9206\n",
    "2. GMF's out-of-sample RMSE is 0.9237\n",
    "3. MLP's out-of-sample RMSE is 0.9094\n",
    "4. NeuMF's out-of-sample RMSE is 0.9059\n",
    "5. Rating prediction out-of-sample performance in terms of RMSE (the lower the better): NeuMF < MLP < ALS < GMF\n",
    "6. The best model is NeuMF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Deep neural networks can achieve a better performance in recommendation system, specifically at Collaborative Filtering\n",
    "2. As you increase the depth of the network or broaden the network branches, the performance of the model also increases "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
